28 May 09 Server Migration: After Action Report
Well the dust has settled and everything has been migrated to the new server. The last big thing to move was surfguitar101.com. I started the DNS change Monday morning, and it was only a few hours later that it had propagated to my house, so I was able to switch on the new site. However, I made a typo in the ServerAlias line in the Apache configuration file, and the alias www.surfguitar101.com wasn’t working (I really wish people wouldn’t use that). I started hearing reports that people still couldn’t get into the site after 8 or 10 hours, but I just figured DNS hadn’t propagated to them yet. This seemed reasonable at the time, because irc.surfguitar101.com took nearly 20 hours to switch for me. But after a day and a half I checked my configuration, and sure enough, oops, a typo. A quick edit and a restart of Apache and everyone was good.
Things over all went very well. I’m just glad I several good books on Apache, DNS and bind, and the Postfix mail transfer agent. And again, it was a great decision to get a second server to migrate to instead of thinking I could just reimage the existing server and get everything back in a few hours.
Since I got everything running I’ve been tinkering around, trying to do a better job of admining this new server. I had a cron job that backed up my databases and SVN repository nightly, but the backups still got stored on the same machine. So I finally broke down and wrote a cool Python script that cron kicks off nightly to transfer the backups to the backup server my host provides. Python is so awesome. I used the Python ftplib to accomplish this. It’s a very useful library, but I thought the interface was kind of odd. You actually have to issue FTP protocol commands as strings, e.g. “STOR.” I think I would have tried to abstract the underlying protocol a bit, but I suppose you get maximum flexibility this way.
I also worked on understanding the System V style startup sequence a bit better. I have several programs that I want to execute as daemons whenever the server is restarted: IRC, my IRC bot, and Teamspeak to name a few. The builtin start-stop-daemon was very useful for this, as well as the script update-rc.d. For my IRC bot, which is written in Python, I found some great code to daemonize it here (thanks Sander!).
And finally, now that I have 3 virtual domains running on the server, I got concerned about all the Apache log files. Each website has both an access log and an error log. On the old server, Plesk had some scheme to rotate these logs. I’m on my own now. But some simple research revealed the already builtin logrotate program. Wow is this useful. I created a simple configuration file and now my logs are getting rotated and compressed weekly. I won’t have to worry about running out of disk space (under normal load anyway).
So with the server situation largely under control, I can finally start working on Django again and getting a beta test site going. More to come.
24 May 09 Changing Server Timezone in Ubuntu
I thought I would jot this down. To change the timzone of your Ubuntu server execute the following command:
$ sudo dpkg-reconfigure tzdata
This Ubuntu community wiki page has some good info on time in Ubuntu.
20 May 09 Server Upgrade Status: Getting There
Welcome, this is my first post from the new server!
I’m really, really glad I went with renting a 2nd server and migrating to it instead of my original plan of just re-imaging. I would have been down for days while I figured out Ubuntu without Plesk.
I got my new server and pressed the magic button to have it re-imaged to Ubuntu 8.04. It all seemed to work just fine. I spent last Saturday going to town on it, getting the version of Subversion I wanted to use built, getting mod_wsgi built, and other simple “new server” type tasks. But then I noticed that the following on Sunday:
/dev/md6 4.7G 228M 4.5G 5% /var /dev/md7 221G 83M 221G 1% /home
The sizes of /var and /home were swapped compared to the old server. I had a huge /var and a small /home before. I became very worried because on the old server, all the databases, website data, and logs all go under /var. Was I going to run out of room very quickly?
I stewed on this for a day or so, and I even toyed with the idea of reimaging to CentOS 5 and getting Plesk back. After all, my main goal was to get Python 2.5, and I subsequently learned you can run different versions of Python side by side. So in theory I could install Python 2.5 on CentOS if I built from source. Then I could tell mod_wsgi to use Python 2.5, meanwhile the rest of the system would still use Python 2.4, and things like the yum package manager would keep working.
But ultimately I decided I liked Ubuntu better. I liked the way they setup Apache for easy multiple virtual hosts. I liked the aptitude program as a simple GUI for apt-get. And Ubuntu is my development environment on my laptop, so having them both match would be a good thing.
I did ask my host if they could either customize the image to make /var bigger or help me do it. They said no to both. This wasn’t totally surprising or unexpected.
I decided it didn’t matter how big /var was. I’ll just put everything on /home, under a directory of my own making. I configured MySQL to use the new directory on /home, and all my virtual host websites will use it as well. Now only system logs will use /var, and it should be big enough for them.
With that settled I set of to try to make it work. I began moving deathofagremmie.com to the new server. Moving the database and WordPress files was easy enough. I then hacked my way through the DNS configuration, using the old server as a guide. I really wasn’t sure I had done it right, but what the heck, I thought I’d give it a try. I then had my host transfer the domain to the new server. There wasn’t much to do except wait for the DNS to propogate. I did this late at night, so I went to bed. The next day at work I noticed that it had transferred, and the website even came up! A few things didn’t work, but nothing I couldn’t fix at home later. But when I came home from work, DNS was still pointing to the old server. I had to wait until almost 24 hours had elapsed before it finally updated for me at home.
Once DNS was correct, I could more easily fix the problems from the migration. It turns out they were minor. I had copy/pasted some Apache configuration from the default files, and I had the “AllowOverride” option set to “none” on my virtual host. This meant Apache would not look for .htaccess files, and thus the mod_rewrite rules that WordPress uses weren’t working, and the authorization I had setup on the admin login was busted. No problem, I’d actually rather not have Apache look for .htaccess files for efficiency reasons. I just moved those directives out of .htacess and put them into my configuration file for the virtual host. Why didn’t I think of that before? But that wasn’t quite the whole story. Mod_rewrite still didn’t work, but that was only because it wasn’t enabled by default in Ubuntu. The Ubuntu team did a really good job of paring down the Apache configuration, and making it very easy to enable and disable modules. A simple “a2enmod” command and a graceful restart fixed it. The Ubuntu Server Guide was a great help to me while I was doing all this.
And yesterday I received my copy of Pro DNS and Bind by Ron Aitchison. By quickly reading the first few chapters the mystery around those DNS files was suddenly solved. I went back and redid those files with my new knowledge and I feel confident about them now. I highly recommend this book.
And, using my new copy of The Book of Postfix, I was able to get my mail aliases for deathofagremmie.com to work. Postfix was much easier to setup than I thought it would be. The developers did a great job of “just making it work” out of the box with minimal configuration. I don’t have any plans to have any real mailboxes on the server, I just want aliases to forward to my Google mail account. This was easy to setup, and is described in Chapter 14 “A Mail Server For Multiple Domains.” I used the virtual alias technique to accomplish this. Edit a few files, run some commands on them, restart and it all worked.
The last wrinkle of the night was figuring out how to change the hostname under Ubuntu. This was solved by doing some quick Googling. You basically have to edit /etc/hostname, /etc/hosts, then stop and start the hostname service. With that fixed, email from my new server looks correct.
So, what’s left? I’m liking this new configuration, so I think I’ll go with it. Now I have to move the two remaining sites: one is a Django powered site, and one is the current PHP based SurfGuitar101.com. After I get them moved and tested I can shut down the old server, and get back to working on the new Django powered version of SurfGuitar101.com.
15 May 09 Server Upgrade, A Slight Return
I started having some pretty serious doubts that I could bring the current server down, get it re-imaged, and then put everything back within a few hours. A major goof or unforseen gotcha could bring the site down for days, which wouldn’t be good for my stress level. So after talking to my old PHP pal Evaders, I decided to follow his recent course of action and rent a second server and then migrate to it at my leisure. This has the added bonus that the latest entry level server packages from my host feature slightly faster processors and double the RAM compared to what I have now. And since I apparently can click through my own affiliate link, it really shouldn’t cost a whole lot. I’ll just be renting two servers for a short period. Once I get everything setup and moved over I can turn off the old one.
So last night I placed an order for another server, and within hours it was ready to go. The first step will be to re-image it from CentOS 5 to Ubuntu 8.04 LTS. Here we go! This should be very interesting.
07 May 09 Firefox, Ubuntu, NVIDIA, and Slow Javascript
I noticed that some web pages that make use of Javascript were acting very slow in Firefox on my Ubuntu 8.04 LTS powered Dell XPS M1530 laptop that has an NVIDIA graphics card. Apparently a lot of other people have too. I finally got annoyed enough to research my options, and a work-around posted in the referenced bug tracker did the trick for me. First I had to do a “apt-get install nvidia-settings” to install a program that lets you tweak the NVIDIA driver settings. Then you need to tweak one setting as follows:
$ nvidia-settings -q InitialPixmapPlacement $ nvidia-settings -a InitialPixmapPlacement=2
The first command shows you the current value of the setting InitialPixmapPlacement. On my system, this was a 1. I wrote this down in case I need to switch it back on a future release. On the second line, you jam that setting to a 2. And voila, everything suddenly ran a lot smoother.
Postscript: Although this did seem to help with some issues, it still isn’t completely resolved. Aargh.
Update 12 November 2009: Changing the Gnome desktop theme to Clearlooks from Human made a noticeable difference. It isn’t perfect yet but it really helped.
Update 2 May 2010: This problem is resolved in Ubuntu 10.04 if you install the nvidia-current package. Use the “Hardware Drivers” program (found under the menu System / Administration) to select “NVIDIA accelerated graphics driver (version current) [Recommended]“. Restart your computer and finally enjoy smooth Javascript.
Tags: firefox, javascript, nvidia, ubuntu
03 May 09 Django-Elsewhere
I just got finished integrating Leah Culver’s django-elsewhere application. Django-elsewhere was formerly Django-PSN (Portable Social Networks) and was originally created for the now defunct social networking site Pownce. This nifty application allows your users to add an arbitrary number of social networks, websites, and instant messengers to their profile. The application even comes with many icons for widely known sites.
In my previous design I had just stuck a few fields in my user profile for websites and a few of the common instant messengers. This was limiting, and I had been thinking about expanding it to a more general solution when I stumbled across this application.
To integrate it with my site, I created a template tag to display a user’s “elsewhere” sites, and I made a view and template to allow a user to edit their sites. This code was based off the example view and template that came with the application. In general the django-elsewhere code quality is quite high. There are still a few print statements in the code base, but that’s all I can find fault with right now.
Thank you django-elsewhere team for the big time saver!
Tags: django, django-elsewhere, sg101
