I prayed

So I’m attached to this website even if it isn’t used much these days, and was torn inside to see it die.

It happens early today (I was going to bed and instead six hours later I’m still up) that I check the website and it gives me a 500 internal server error. So I go to the Dreamhost page to report the problem but I soon notice that I can access the databases and FTP. So it’s not down, it’s broken. I go to the support page and I see a warning stating that they are moving the site on a different server and if I had a custom PHP (I have, because they already broke my site once and I had to resort to that) then it was going to be broken after the move.

The problem is that I don’t remember anything about how I actually made the custom PHP. Not even a vague idea. The perspective was starting to feel rather gloomy. But then looking at the root of the site I notice that I left all the stuff still there and it’s actually working, including the old script I used and modified (because it also didn’t work right away back then). Only that even my script doesn’t work now (I’ll explain the tech details later).

And there starts the odyssey, looking through all kinds of websites, copy/pasting errors and whatnot. I rarely made a so hectic journey through the internet. The old engine just didn’t seem compatible with the new server and wouldn’t compile. In six hours, without knowing any damn think about all this, I think I tried a hundred of different solutions, editing scripts and makefiles for the compiler even if I never really saw one before.

Well, I’m happy. Now it works again. Dunno if it doesn’t break suddenly at some point because there are still some tricky things, but at least I got to see my site once again. It was truly so close to being gone and I was hopeless because Dreamhost simply told to deal with it and that they would not offer any support of any kind.

Later I’ll have to go and documenting this a bit, I’ll do my part and also update the broken wiki page at Dreamhost.

I’m not really sure I want to go down in detail to explain what I did because I don’t want to make the site even less secure and broken. Here’s the essence of what happened:
The website runs on custom compiled PHP. The website gets moved to a 64bit machine and so all the old code is now broken. I need to recompile.

I still have everything from when I did compile PHP last time, so I try to rerun the same script. But it breaks:

configure: error: Cannot find OpenSSL's <evp.h>

Apparently Dreamhost doesn’t have anymore support for OpenSSL and PHP requires it. Where things really start to get gloomy is when you look for the error online and find other Dreamhost users who faced the same problem and asked in forums and blog posts… years ago and unanswered. The internet reveals the desperation of passing time and unanswered calls for help! That’s the kind of ending that I was looking at. Doomed.

So I try to look for the source code and compile it locally, which isn’t simple, because adding it to the other script didn’t work. The “configure” command didn’t work, I read that it used “config” instead, but it still wouldn’t run properly. Follow an endless number of attempt and I think I was able to finally run it through “sh”, the shell.

Then I had to make PHP know where to find OpenSSL locally, which required a lot of juggling of directories (while also many attempts to see if I could compile PHP without SSL support, maybe) which is hard because you don’t know if it doesn’t work because pointed in the wrong location or because it isn’t compatible or misses some part. At the end OpenSSL works but the script breaks again two checks after:

/usr/bin/ld: cannot find -lc-client
skipping incompatible /usr/lib/libc-client.a when searching for -lc-client

Fun! I later figure out that the lc-client is compiled in the IMAP module and copied over. I tried compiling a more recent version (but the problem was elsewhere) but that lead to a whole host of new problems since I was getting incompatibilities with IPV6 and the program attempting to overwrite (and failing) some main files on the server. Then it turned out that IMAP also required OpenSSL, but it wasn’t smart enough to look at the local copy I installed (nor had a configurable “config”), and so continued to fail. So I tried to get my hands down deep in the makefile itself, trying to link manually my OpenSSL, but I only got more and more errors and screen showing fancy characters. No good at all. I tried with different version of IMAP hoping maybe one would be compatible.

This goes on a while, with my root directory filled with sss aaa abba azazphp aaaphp and so on because every run of the script pretended I erased the directories and doing it through FTP would take an insane amount of time and I didn’t remember the Unix command that would delete a directory with everything inside (rm -r), and I was too busy with a million of other thoughts to look that up.

In the end I got through, managed to compile IMAP and have it digested by PHP. And the site ran again.

I also suspect that lots of troubles also came from the possibility of having code half compiled and uncleaned since I compiled PHP last time.

(this omitting many steps in the middle, like installing other libraries that may be connected and swapping source versions with newer ones)

Posted in: Uncategorized | Tagged:

Leave a Reply