Eccentric Flower:201007/How To Not Install

Warning: This entry contains huge amounts of technical gibberish - although understanding exactly what is happening is less important than getting the general sense of the cautionary tale.

Monday

The problem: Your office wiki, which most people in the office don't even use because wikis scare them and they would rather use SharePoint because they are masochistic and insane because it allows them simply to drop in Word documents rather than having to actually use the wiki's editor, whereas you don't work in Word unless forced to at gunpoint but the wiki's editor is second nature to you, so despite your having gone to great pains to set it up three years ago at your boss's request, you are virtually the only person who keeps any serious amount of information in it, but for you it's absolutely indispensible -

Sorry, let's try that again. So

The problem: The wiki is currently on a Solaris VM (see ** way below) on a machine your boss would like to retire; therefore it needs to be moved to a VM on a machine that you will be keeping around for a while. Virtually all your servers are Solaris 10 VMs (Solaris calls them "zones") because they can be created, destroyed, and cloned relatively easily - by those who know how. Unfortunately the cloning part only works if both VMs in question are on the same physical server, which these will not be, because that's the whole point here.

So you are given the keys to a VM which was set up for some forgotten purpose by your ex-sysadmin some years back and never actually used. This is the equivalent of inheriting an old house full of dust and cobwebs - you have no idea how far she got or what's lurking in there, but it doesn't matter because you can tear it all out if need be. Anyway, anything she did was likely to have been pretty minimal, since she was a fairly faithful recorder of her build processes and there are no notes from her at all on this VM. Nothing.

To set up MediaWiki on this new server, you will need first to install MySQL, without which MediaWiki is nothing. MediaWiki runs with other databases, sometimes, with a great deal of cajoling and voodoo, but it runs with MySQL fairly painlessly.

You will also need to install a program called phpMyAdmin, a graphic UI for MySQL, because administering MySQL from the command line is a black art and one you do not care to practice, especially since you are an Oracle pro and feel that MySQL does many things The Wrong Way and verily it confuseth you. (No one has, to your knowledge, ever even bothered to try to get MediaWiki to cooperate with Oracle. This is because MediaWiki was written, and is predominantly used, by the kind of people who would invite you to put hot spikes under their fingernails before they paid for an Oracle license.)

Of course, as the name suggests, phpMyAdmin is a PHP-based set of scripts, so you will have to also install the demon PHP. (You'd have to anyway; MediaWiki requires PHP also.) Perl is routinely installed on every server you touch, but PHP is installed only on the rare machines which have need for it - which, in practice, means the two or three machines which have reason to use MySQL - to wit, the old wiki server, this new wiki server, and your colleague's various Drupal and Wordpress servers which she uses to meet the constant 'we want an instant disposable weblog' demands from your users. You hate both Drupal and Wordpress, but you admit that giving them an installation like that is a much better solution than giving them a set of pages and name redirects and possibly Perl scripts on your main web servers, especially since your users tend to set up projects and then abandon/forget them after a month, and you are in the business of long-term scripting for your real web services - that is, web-based front ends to serious Oracle DB tasks - and can't be bothered to write a set of Perl CGIs for whatever flavor-of-the-month project comes along (unless it's one of yours). It would be great if you could maintain the Perl/Oracle machines and she could maintain the odious PHP/MySQL machines, but for some reason, Perl seems to be less and less in demand even though it kicks PHP's ass, so you are resigned to your fate.*

* My tongue is more than slightly in my cheek here. Actually my attitude at the workplace has always been "whatever is the most effective tool to get the job done," and my only issues with PHP are that I don't like the fact that it has absolutely no separation between "code" and "page content," and its syntax is a little too Javalicious for me in some ways. It's more useful than Perl for some things, less useful than Perl for others. What I don't understand is why, when both of them share essentially the same strengths and the same faults, and both are equally easy to abuse in horrendous ways, why do so many people look down their nose in horror at Perl but don't have the same kneejerk bad reaction to PHP? If anything I think it's slightly easier to write bad PHP than bad Perl. Is this just because Perl's been around so much longer and familiarity has had time to breed contempt?

When getting the MySQL package, you notice that they are trying very hard to prevent people from using old versions, and the PHP site says the same thing, but that's okay, because you don't see any reason why you shouldn't just go with the most recent version of each.

Looking at the main PHP site you see that they do not distribute binaries - not a surprise - and your choices for Solaris binaries are either from OpenCSW, which you've never used, or from SunFreeware, which you are well familiar and comfortable with. The CSW package is bare-bones and requires you to download and build in a whole slew of other packages and you really think it's more trouble than it's worth, especially since SunFreeware has a package which already has everything you'll need included. The only problem is that their PHP package which is compatible with the MySQL you will be downloading also requires you to use Apache 2.2.15, not 2.0.x.

Come to think of it, which Apache is running on this dusty VM anyway?

You inspect the process list and you find that it's running Apache 1 out of a wholly unexpected directory. Not only that, but there are no fewer than six different Apache installations on the system, and you don't trust any of them. You decide to eradicate them all since you're going to need to install a newer version anyway. This means you'll also need to edit the system startup scripts so that when the machine is rebooted, it will restart the correct Apache in the correct place with the correct configuration files. You start to edit the files -

- and you realize that there is no emacs on the system. Your ex-sysadmin, who unlike you was a greatly opinionated person, was proficient in vi and a little scared of emacs because she was masochistic and insane. Thus, while she always put Perl on every system because she knew you'd be lost if it wasn't there, she resisted putting emacs on any systems even though she knew you'd be lost if it wasn't there. "You should learn vi," she said, "it's always installed and one day you're going to be stuck on a system you can't put emacs on." Personally, you'd download the files to a machine that does have emacs and edit them there before you'd use vi, an editor which proves that far too many old-timers in the Unix community were raised to consider masochism a virtue.

So you stop to install emacs, which must be compiled from scratch because if someone is distributing Solaris 10 binaries of emacs, you have never found it. This isn't too big a pain; you've compiled emacs more times than you can count. However, your ex-sysadmin also didn't believe in installing any sort of useful compiler environment unless forced to, and this system keeps trying to look for a home directory for you that doesn't exist (which also means that you're operating without tab-completion or a command buffer, which means when you get a typo in a thirty-character-long package or tarfile name you have to type the whole command again, which is not helping your mood or your productivity a bit), and it has no decent path settings, so first you have to find where gcc is; then you have to find where make is (it's in an entirely different place); then you have to hope they work well together even though they're obviously from completely separate installations (there will be more on this below); then you get started and the emacs compilation objects to some things that you really didn't even want it to compile in the first place; and eventually you realize that you have to tell its configuration tool about seventeen different parameters for "don't give me this kind of graphics support, don't give me that kind of graphics support, don't give me ANY kind of graphics support, don't give me X support, who the hell told you to automatically compile in X anyway, don't give me mouse support, there is no mouse here, don't give me any of that crap, just compile a damned text editor."

Once you do that (and stop to gripe on Twitter about how X support should always be optional because Unix is not a graphical system and you have used X maybe five minutes of your life despite working on and around Sun boxes for twenty years; Unix is a text-based, command-line OS), you do eventually get emacs to compile, but it won't fully install because it keeps choking on setting rights (which are already set correctly) for input method libraries, most of which are damned Asian support you'll never need and you wish you could have opted out of anyway, and finally you just go in and edit the damned makefile for that section of the install (using emacs, which is now working just fine, proving what a waste of time this is) just so the make install won't stop dead there and can go on copying the rest of the crap. You pause to reminisce fondly of a time when emacs was not bloated with useless fripperies.

Now it is time to install the Apache package, which you do. Then you go into the rc directories and init.d and deal with the startup sequences, which, fortunately, go exactly as expected. You lose close to an hour reconciling all the various versions of Apache configuration files all over the system and removing all the false ones so there will never again be such confusion. And then, the moment of truth. Having killed the old running one, you try to start the new Apache.

It is at this point that you discover that, even though this is a packaged binary, set-and-forget, which pkgadd installed without a hitch, SunFreeware has done you a misservice. Normally when you add a package it reports any missing dependencies. But the Apache package did not; it only fails on its missing dependencies at run time. And while you know what dependencies it needs, you have no idea which are already present.

Rather than spending your next hour embroiled in quality time with the find command, you instead do this:

• Attempt to start Apache.
• Note the library name that fails.
• Go back to SunFreeware, get that library package, install it.
• Attempt to start Apache.
• Note the next library name that fails.
• Go back to SunFreeware, get that library package, install it.
• Lather, rinse, repeat.

Four packages later, you have a working Apache 2.2 installation. Life is good. Except then it's after four-thirty and you haven't even begun the main course of the day's installation work. (Mind you, you didn't start until after noon. Other business of the day first.)

Oh, by the by, you're getting all these packages and such from the web, so the process for each one is

• Download them to your computer;
• then re-upload them to the server via sftp.

In essence, each file must be transmitted twice. Fortunately they're all reasonably small except MySQL, which is ginormous.

Speaking of MySQL, you figure you can at least get a start on unpacking it. You untar it, but you notice that tar is doing some strange things. Apparently some filenames are very long and it doesn't know how to handle them. You look in the README file in the MySQL directories you've just unpacked, and it explains that older versions of tar do choke on this, and that you should instead use the gnu gtar - which, thank god, is already installed, so all you have to do is delete everything you just unpacked and untar it all again, this time using gtar instead.

You then read the set of steps needed to install MySQL - which is a mile long and that's for a binary installation, no compilation involved - and you decide to write this entry up to this point instead. By the time you get done with this much of the ranting, it is 5:45, and you decide to continue bright and early the next morning - maybe stopping to give yourself a proper home directory first because you're tired of emacs fussing about that each time you try to edit a file.

Tuesday

You begin the day by setting up a proper home directory and also changing the default paths, something you'd never done before. You are not actually a highly-travelled Unix sysadmin - for most of your working life you have depended on other people's Unix installations and sysadminning, so there are quite a few things you learn how to do only when you need to do them and no one else is around to do them. For example, you always thought that 'su' was a command to become root user, but today you have learned that it's really a command for assuming the identity of any other user on the system - assuming you have the rights to do so. And now its peculiarities about which settings it uses at any given time make sense, which they never did before. So that starts the day on a good note.

To your surprise, the MySQL binary installation goes flawlessly, with some well-placed tips which pop up exactly when you need them to remind you what to do next, and a very nice "make the installation secure and delete all the test cruft" script which does exactly what it's supposed to. It even provides you with a ready-to-use script to put into init.d so you can tell the system to start the mysql service automatically upon boot.

These will be the high points of your day.

Once MySQL is up and running, you proceed to PHP. The PHP you are planning to use is a precompiled binary. You expect it will probably fail on a few missing dependencies, and it does. The first one it fails on is a library called libsybdb.so.5. Now, Solaris library names are generally fairly consistent with what they do. If you are missing, for example, libcurses.so, then you know you need to install the 'curses' package; if you're missing libintl.so you need an 'intl' package, and so on. But the missing library in this case - which looks, to trained eyes, like some sort of Sybase DB support - turns out to be in a package called 'freetds,' where the 'tds' stands for Tabular Data Stream. It is Sybase related, but that is no consolation for the fifteen minutes you lose searching about it on the web.

Then you hit a bigger problem. PHP seems to be looking for libnsl.so.1 - except in this case there is one of those on the system, and it's the wrong version. Furthermore, this seems to be something that you can't get in an add-on package; it's a Solaris core library. Furthermore (you determine after a half hour of research) there is nothing wrong with your library and replacing it could be a very bad idea, and the problem may well be a bug that you have no way to fix except to build PHP yourself. Well, if you can't get past this point, the next option is to build it yourself anyway.

You do learn one handy trick, though, which is that there is a command 'ldd' which you can run on a binary, and it will tell you exactly what versions of what libraries that binary expects to find, and where it's finding them - which would have greatly simplified the multi-step Apache "lather, rinse, repeat" process described yesterday.

Eventually it becomes clear that you have no choice but to try to build PHP from scratch. You spend some time very carefully reading through the code's many configuration switches to make sure you get exactly the ones you want, and fiddling with paths to make sure it will find everything it needs. Most of what you want to do with configuration is exclude things you don't need. You pause for a moment to gripe about the expectations of idiot users who file bug reports if their everything-but-kitchen-sink options aren't automatically included in the compilation, the people who compile without reading the switches and just expect everything to be there. The default, you reflect, should be to include nothing - all options should be opt-in rather than opt-out. But no, then people would whine and moan; so once again your life must be made difficult so that everyone else's can be made lazy.

You are possibly feeling a wee bit martyred at this point.

You attempt to compile. It can't find certain things that you know you put in the path. You check the web. You read about the switches some more. You repeat by trial and error several more times. Finally you get it so it can find everything it needs to find. Then the linker fails, spitting out a list of hundreds of unreferenced symbols.

You come to realize, thanks to a few helpful web pages, that there are two basic problems. One of them is an old favorite.

Very few people ever use the native Solaris compiler (cc) because the GNU compiler (gcc) is better. But for some reason that you have never been able to fathom, the normal procedure for building the GNU compiler on Solaris is to build it using the Solaris linker (ld). This is a problem because many other pieces of software, when they are being compiled, ask the compiler "What ld was used to build you?" and it tells them and that's the ld they want to use. Unfortunately, the Solaris linker often doesn't work well when used in certain ways in conjunction with the GNU compiler. It would make far more sense to use the GNU linker to build the GNU compiler. The problem, as far as I can tell, is that in order to compile the GNU linker you have to use the GNU compiler, but you can't make the GNU compiler without a linker! In short, the reason gcc is so often built with the Solaris ld is that the Solaris ld is already there, sitting around, and thus you can avoid this whole chicken-and-egg business.

But eventually it comes back to bite you. The solution, of course, is to go get the GNU support tools, including the proper ld. First you try to build them from scratch because that's the only way the GNU site will give them to you, as source. This loses you nearly an hour and much heartache and pain. Eventually you go to SunFreeware and get a package and just install that. Then you have to be very careful about your paths, because now your system must be able to find both ld's - the Solaris one and the GNU one - but must always find the GNU one preferentially, which means your path must be arranged just exactly so.

Once this is taken care of, you still end up with a lot of bad linker symbols. Incidentally, these attempted compilation runs are not instantaneous; a great deal of your day is spent with an upset stomach watching a long slow compilation process slog through the part you already know works, while you worry what will happen this time when it gets to the part which has failed.

The other issue apparently seems to be that the MySQL package available from the official MySQL site does not come with shared libraries. If you want shared libraries, you have to compile them yourself. You may ask yourself, "What good is a pre-compiled, binary installation if it doesn't have shared libraries?" Certainly your wife instantly asks this question when you describe the situation, and your wife, while no novice, is hardly a Unix pro. The incompetence level is so evident not even a child could miss it.

So you download the MySQL source code package and build it. Mind you, you already have an existing MySQL installation that you're not sure you want to step on yet, so you build the new one in a different location. This build takes a very long time. And you have to do it twice, because the first time it chokes on a missing dependency near the end. (All compilations choke near the end. They never choke right at the beginning and save you the wait.)

Back to the PHP build. Now it gets past the linking stage - hooray! - but then fails because it can't find a MySQL library that you deliberately didn't compile because you didn't see why you needed to recompile the MySQL server from scratch, just the shared libraries. But apparently the PHP code wants to see 'mysqld' (that's the server daemon) for some inexplicable reason, so you go back and change the MySQL compilation switches a little and begin building the whole damned thing again. Longer, even, this time, because now you are building server and everything from scratch.

As of 5:18 this compilation is still running, you have played four different DS cartridges during compilation downtimes until you are bored with each of them, your day is shot, you still haven't gotten anywhere near the MediaWiki installation that was the point of all this, and you're contemplating whether it is possible to take gin intravenously. And lying painfully at the bottom of your stomach is the suspicion that once this compilation finishes, the PHP compilation still won't work.

As it turns out, the matter is moot because, after nearly an hour's compilation, the full compilation fails - when trying to make 'mysqld,' the one thing you really need from it.

Fortunately, you toss the text of that error into the web - god bless the web, because any error you get anywhere in any of these processes, someone else has already gotten it and written a page or a bug report about it - and you find that it has to do with Solaris stupidity about the linker and empty files, related to how MySQL has a plugins mechanism but does not start off with any plugins; and you add the following two lines into a conditional directive of one source file:

#else
int Sun_ar_requires_a_placeholder_here= 0;

and MySQL compiles successfully. It is now 5:49. Installing the freshly-made files will probably take another ten minutes. You have vowed that, whatever condition this mess is in by 6:20, you are stopping for the night then. You have beaten your best time on six tracks of Mario Kart DS time trials so far.

You continue to get a failure when PHP tries to link the 'mysqld' it shouldn't have been looking for in the first place. You trace the logic backward and solve that problem by removing a "try to do this internally to PHP" switch which, you determine, was attempting to try to call the MySQL daemon remotely to do SQL rather than the old-fashioned way. Would have been a nice feature to have, but if taking it out means this beast will compile, so be it. (Also means you wasted that whole second MySQL compilation since you have now removed the need for it, but oh well.)

You get past that point in the compilation. Hooray! Then you end up with a list of unresolved linker symbols again, this time having to do with the improved MySQLi syntax. You turn off MySQLi. Again, would have been a nice feature, but you can't waste any more time on this compilation.

This time, when you compile, you get a different two unresolved symbols, this time in the basic MySQL libraries. Those you can't turn off. You are essentially right back where you started before you built MySQL from scratch. You have no idea how to proceed, your day is shot, and you are on the verge of either breaking into nervous tears or smashing something.

Wednesday

You need to physically go into the office (you're working at home a lot lately because your offices are being torn apart and remodeled, and the temporary work space is unsuitable for concentration on anything) to attend meetings. Catching up with your boss, you describe the past two days and the agonies therein.

Your boss says, "Let's just go get a Linux box."**

** Your boss reminds you in this conversation that the old wiki was actually on a Linux VM - that in fact all the PHP-application servers you have are on Linux machines. This erases your "How the hell did we ever get this running on the old machine?" questions. Of course it was easier there - it wasn't Solaris. Solaris just doesn't seem to do well with the MySQL/PHP stack. More comments on this below.

You say, "Is external hosting okay?" You and your boss are experimenting with gradually moving away from university hosting, which is expensive and difficult, to third-party hosting on commercial services.

He says, "Go for it. Want my purchasing card?"

You get a business account with the same hosting provider that runs this site. Five minutes after you hand them some cash, the email arrives with a printable copy of your invoice and your passwords and your control panel access and your temporary URLs.

You don't get a chance to actually work with the site until Thursday, at which point you go to the control panel and press the button that says, in essence, "Install MediaWiki now." The installation-complete page shows less than five seconds after you press the button. You go to the new URL and, lo, there is a working wiki. The database still needs to be duplicated from the old site, and a lot of customization needs to be done, but the fact remains that in less than ten minutes of total overhead, you have what you tried, unsuccessfully, for two days to do elsewhere.

The Moral

I've been using Sun/Solaris systems more or less exclusively for my entire Unix career. This is very unusual, and is a historical accident of my employment path. I don't hate Solaris. There are many features of it which are very nice and useful - the zone/VM management is extremely handy if you're trying to run a lot of different small servers, for example. Also, since the basic OS is maintained by a company that actually has an interest in producing a consistent product - or at least used to be - and not by a bunch of open-source hackers whom I distrust - the theory is that the Solaris OS will work better than open-source OSes. And it does - if you limit yourself to considering just the bare OS.

But this is far from the first time I have encountered problems with installing and running additional software - and, let's face it, the "additional software" is what makes the world go 'round. I very seldom have to think about the internals of the Solaris OS, but I spend my entire day working with things like Perl and PHP and MySQL and MediaWiki and Apache.

As far as I can tell, if you install a set of packages across the board, from the applications on down to the tools, that are more or less of the same age and have been verified to play well together, then it all works. In other words, if you were willing to keep every single package on your machine at a consistent version level, you'd be fine. But this would mean that when you wanted to upgrade MediaWiki you'd also have to upgrade everything else, maybe as far down as your compiler and some of your OS tools. No one does that. People upgrade things piecemeal; and far too often, when you try piecemeal upgrades on Solaris, the whole house of cards falls apart.

I'm not paid to be in the software-compilation business. My boss, upon hearing the saga, concluded instantly - and I agreed readily - that those two days were not a good use of my time. The thing is, even among people who are paid to install and compile and wrangle software, only so much of this sort of wrestling match can be tolerated. People eventually vote with their feet and change to a system that Just Works more often.

This does not bode well for the future of Solaris, I think. I am not a hardcore Unix admin and I never will be, but I've been doing this sort of installation dance for years now; and I think I am qualified at this point to say that Solaris doesn't Just Work nearly as often as it should. Whether because the OS is a delicate flower, or because of poor support from software makers and others in the community, I don't know. Nor do the reasons why matter very much.