28 May 09 Server Migration: After Action Report

Well the dust has settled and everything has been migrated to the new server. The last big thing to move was surfguitar101.com. I started the DNS change Monday morning, and it was only a few hours later that it had propagated to my house, so I was able to switch on the new site. However, I made a typo in the ServerAlias line in the Apache configuration file, and the alias www.surfguitar101.com wasn’t working (I really wish people wouldn’t use that). I started hearing reports that people still couldn’t get into the site after 8 or 10 hours, but I just figured DNS hadn’t propagated to them yet. This seemed reasonable at the time, because irc.surfguitar101.com took nearly 20 hours to switch for me. But after a day and a half I checked my configuration, and sure enough, oops, a typo. A quick edit and a restart of Apache and everyone was good.

Things over all went very well. I’m just glad I several good books on Apache, DNS and bind, and the Postfix mail transfer agent. And again, it was a great decision to get a second server to migrate to instead of thinking I could just reimage the existing server and get everything back in a few hours.

Since I got everything running I’ve been tinkering around, trying to do a better job of admining this new server. I had a cron job that backed up my databases and SVN repository nightly, but the backups still got stored on the same machine. So I finally broke down and wrote a cool Python script that cron kicks off nightly to transfer the backups to the backup server my host provides. Python is so awesome. I used the Python ftplib to accomplish this.  It’s a very useful library, but I thought the interface was kind of odd. You actually have to issue FTP protocol commands as strings, e.g.  “STOR.” I think I would have tried to abstract the underlying protocol a bit, but I suppose you get maximum flexibility this way.

I also worked on understanding the System V style startup sequence a bit better. I have several programs that I want to execute as daemons whenever the server is restarted: IRC, my IRC bot, and Teamspeak to name a few. The builtin start-stop-daemon was very useful for this, as well as the script update-rc.d. For my IRC bot, which is written in Python, I found some great code to daemonize it here (thanks Sander!).

And finally, now that I have 3 virtual domains running on the server, I got concerned about all the Apache log files. Each website has both an access log and an error log. On the old server, Plesk had some scheme to rotate these logs. I’m on my own now. But some simple research revealed the already builtin logrotate program. Wow is this useful. I created a simple configuration file and now my logs are getting rotated and compressed weekly. I won’t have to worry about running out of disk space (under normal load anyway).

So with the server situation largely under control, I can finally start working on Django again and getting a beta test site going. More to come.

Tags: daemon, ftplib, logrotate, migration