19 Sep 09 Server Monitoring with Munin
I finally got around to investigating server monitoring tools, and came across Munin. This tool allows you to monitor various nodes in your network, and it generates pretty graphs for a wide variety of system parameters. You install munin-node software on the machines you wish to monitor. The node software gathers statistics which are collected periodically by a munin master agent, which produces the nice graphs that you can view with a web browser.
Installing it was easy with Ubuntu, as packages already exist for it. It was a simple “sudo apt-get install munin munin-node.” The Ubuntu package set me up for running both the master and node software on the same machine. I then had to edit the /etc/munin/munin.conf to change a few paths around, notably where to put the generated HTML files. One hitch that stumped me for a few minutes was a permissions problem on the path to the location of the HTML files. The Ubuntu package creates a new munin user, so make sure all directories on the path to the HTML files have an “x” bit for world. Once that was sorted I started seeing data after 5 minutes for a variety of things including CPU load, MySQL statistics, and filesystem usage. Awesome! I decided to put Apache Basic Authorization password protection on the directory where the report files live, as it is sensitive data that I wouldn’t want just anyone to see.
Below is an example graph, one of about 50 (!) that are available right out of the box. Plugins exist and can be created for a wide variety of other parameters as well. In addition, Munin can notify you by email if parameters get out of range. See the Munin home page for more examples and information.
Munin MySQL Graph
Munin looks like a very useful tool to keep an eye on your server. Thanks to the Munin team for this great piece of software!
Tags: monitoring, munin, Server, tools
04 Sep 09 Denyhosts: SSH Brute Force Protection
I’m always very uneasy when I look through my web server and system logs. It seems dozens of people (or more!) a month try to log into my server, attempt an SQL injection attack on one or more of my hosted websites, and probe for weaknesses. It really makes me lose some faith in humanity.
I was tipped of to the Denyhosts script via a tweet from @dozba. This is nothing more than a Python script that analyzes your ssh log file, looking for suspicious activity. It monitors the number of failed SSH login attempts over time, and once a configurable threshold is reached, it adds the IP address of the remote host to your hosts.deny file, thus blocking the brute-force break-in attempt. The script is very configurable. It can have different thresholds for login attempts to existent or non-existent user accounts, it can purge blocked entries, and it will email you when it blocks a host.
Installation and configuration on Ubuntu is a breeze, as it is available as a package. Use your favorite package manager and install it. Ubuntu automatically configures it to run in daemon mode, so you won’t need to mess with cron. Configure it by editing the file /etc/denyhosts.conf. Restart the daemon, and then you are done. In a nutshell:
$ sudo apt-get install denyhosts $ sudo vi /etc/denyhosts.conf $ sudo /etc/init.d/denyhosts restart
Two days after I installed it, I had already blocked 5 break-in attempts. Thanks to the author Phil Schwartz and the Ubuntu package maintainers for this very useful utility!
28 May 09 Server Migration: After Action Report
Well the dust has settled and everything has been migrated to the new server. The last big thing to move was surfguitar101.com. I started the DNS change Monday morning, and it was only a few hours later that it had propagated to my house, so I was able to switch on the new site. However, I made a typo in the ServerAlias line in the Apache configuration file, and the alias www.surfguitar101.com wasn’t working (I really wish people wouldn’t use that). I started hearing reports that people still couldn’t get into the site after 8 or 10 hours, but I just figured DNS hadn’t propagated to them yet. This seemed reasonable at the time, because irc.surfguitar101.com took nearly 20 hours to switch for me. But after a day and a half I checked my configuration, and sure enough, oops, a typo. A quick edit and a restart of Apache and everyone was good.
Things over all went very well. I’m just glad I several good books on Apache, DNS and bind, and the Postfix mail transfer agent. And again, it was a great decision to get a second server to migrate to instead of thinking I could just reimage the existing server and get everything back in a few hours.
Since I got everything running I’ve been tinkering around, trying to do a better job of admining this new server. I had a cron job that backed up my databases and SVN repository nightly, but the backups still got stored on the same machine. So I finally broke down and wrote a cool Python script that cron kicks off nightly to transfer the backups to the backup server my host provides. Python is so awesome. I used the Python ftplib to accomplish this. It’s a very useful library, but I thought the interface was kind of odd. You actually have to issue FTP protocol commands as strings, e.g. “STOR.” I think I would have tried to abstract the underlying protocol a bit, but I suppose you get maximum flexibility this way.
I also worked on understanding the System V style startup sequence a bit better. I have several programs that I want to execute as daemons whenever the server is restarted: IRC, my IRC bot, and Teamspeak to name a few. The builtin start-stop-daemon was very useful for this, as well as the script update-rc.d. For my IRC bot, which is written in Python, I found some great code to daemonize it here (thanks Sander!).
And finally, now that I have 3 virtual domains running on the server, I got concerned about all the Apache log files. Each website has both an access log and an error log. On the old server, Plesk had some scheme to rotate these logs. I’m on my own now. But some simple research revealed the already builtin logrotate program. Wow is this useful. I created a simple configuration file and now my logs are getting rotated and compressed weekly. I won’t have to worry about running out of disk space (under normal load anyway).
So with the server situation largely under control, I can finally start working on Django again and getting a beta test site going. More to come.
24 May 09 Changing Server Timezone in Ubuntu
I thought I would jot this down. To change the timzone of your Ubuntu server execute the following command:
$ sudo dpkg-reconfigure tzdata
This Ubuntu community wiki page has some good info on time in Ubuntu.
20 May 09 Server Upgrade Status: Getting There
Welcome, this is my first post from the new server!
I’m really, really glad I went with renting a 2nd server and migrating to it instead of my original plan of just re-imaging. I would have been down for days while I figured out Ubuntu without Plesk.
I got my new server and pressed the magic button to have it re-imaged to Ubuntu 8.04. It all seemed to work just fine. I spent last Saturday going to town on it, getting the version of Subversion I wanted to use built, getting mod_wsgi built, and other simple “new server” type tasks. But then I noticed that the following on Sunday:
/dev/md6 4.7G 228M 4.5G 5% /var /dev/md7 221G 83M 221G 1% /home
The sizes of /var and /home were swapped compared to the old server. I had a huge /var and a small /home before. I became very worried because on the old server, all the databases, website data, and logs all go under /var. Was I going to run out of room very quickly?
I stewed on this for a day or so, and I even toyed with the idea of reimaging to CentOS 5 and getting Plesk back. After all, my main goal was to get Python 2.5, and I subsequently learned you can run different versions of Python side by side. So in theory I could install Python 2.5 on CentOS if I built from source. Then I could tell mod_wsgi to use Python 2.5, meanwhile the rest of the system would still use Python 2.4, and things like the yum package manager would keep working.
But ultimately I decided I liked Ubuntu better. I liked the way they setup Apache for easy multiple virtual hosts. I liked the aptitude program as a simple GUI for apt-get. And Ubuntu is my development environment on my laptop, so having them both match would be a good thing.
I did ask my host if they could either customize the image to make /var bigger or help me do it. They said no to both. This wasn’t totally surprising or unexpected.
I decided it didn’t matter how big /var was. I’ll just put everything on /home, under a directory of my own making. I configured MySQL to use the new directory on /home, and all my virtual host websites will use it as well. Now only system logs will use /var, and it should be big enough for them.
With that settled I set of to try to make it work. I began moving deathofagremmie.com to the new server. Moving the database and WordPress files was easy enough. I then hacked my way through the DNS configuration, using the old server as a guide. I really wasn’t sure I had done it right, but what the heck, I thought I’d give it a try. I then had my host transfer the domain to the new server. There wasn’t much to do except wait for the DNS to propogate. I did this late at night, so I went to bed. The next day at work I noticed that it had transferred, and the website even came up! A few things didn’t work, but nothing I couldn’t fix at home later. But when I came home from work, DNS was still pointing to the old server. I had to wait until almost 24 hours had elapsed before it finally updated for me at home.
Once DNS was correct, I could more easily fix the problems from the migration. It turns out they were minor. I had copy/pasted some Apache configuration from the default files, and I had the “AllowOverride” option set to “none” on my virtual host. This meant Apache would not look for .htaccess files, and thus the mod_rewrite rules that WordPress uses weren’t working, and the authorization I had setup on the admin login was busted. No problem, I’d actually rather not have Apache look for .htaccess files for efficiency reasons. I just moved those directives out of .htacess and put them into my configuration file for the virtual host. Why didn’t I think of that before? But that wasn’t quite the whole story. Mod_rewrite still didn’t work, but that was only because it wasn’t enabled by default in Ubuntu. The Ubuntu team did a really good job of paring down the Apache configuration, and making it very easy to enable and disable modules. A simple “a2enmod” command and a graceful restart fixed it. The Ubuntu Server Guide was a great help to me while I was doing all this.
And yesterday I received my copy of Pro DNS and Bind by Ron Aitchison. By quickly reading the first few chapters the mystery around those DNS files was suddenly solved. I went back and redid those files with my new knowledge and I feel confident about them now. I highly recommend this book.
And, using my new copy of The Book of Postfix, I was able to get my mail aliases for deathofagremmie.com to work. Postfix was much easier to setup than I thought it would be. The developers did a great job of “just making it work” out of the box with minimal configuration. I don’t have any plans to have any real mailboxes on the server, I just want aliases to forward to my Google mail account. This was easy to setup, and is described in Chapter 14 “A Mail Server For Multiple Domains.” I used the virtual alias technique to accomplish this. Edit a few files, run some commands on them, restart and it all worked.
The last wrinkle of the night was figuring out how to change the hostname under Ubuntu. This was solved by doing some quick Googling. You basically have to edit /etc/hostname, /etc/hosts, then stop and start the hostname service. With that fixed, email from my new server looks correct.
So, what’s left? I’m liking this new configuration, so I think I’ll go with it. Now I have to move the two remaining sites: one is a Django powered site, and one is the current PHP based SurfGuitar101.com. After I get them moved and tested I can shut down the old server, and get back to working on the new Django powered version of SurfGuitar101.com.
15 May 09 Server Upgrade, A Slight Return
I started having some pretty serious doubts that I could bring the current server down, get it re-imaged, and then put everything back within a few hours. A major goof or unforseen gotcha could bring the site down for days, which wouldn’t be good for my stress level. So after talking to my old PHP pal Evaders, I decided to follow his recent course of action and rent a second server and then migrate to it at my leisure. This has the added bonus that the latest entry level server packages from my host feature slightly faster processors and double the RAM compared to what I have now. And since I apparently can click through my own affiliate link, it really shouldn’t cost a whole lot. I’ll just be renting two servers for a short period. Once I get everything setup and moved over I can turn off the old one.
So last night I placed an order for another server, and within hours it was ready to go. The first step will be to re-image it from CentOS 5 to Ubuntu 8.04 LTS. Here we go! This should be very interesting.
29 Apr 09 Server Migration Is Go…
Well I’ve been doing some reading and thinking, and I think I have mentally prepared myself to re-image the server with Ubuntu 8.04 LTS. Everything seems to be manageable without Plesk. I did some reading, and discovered the default Mail Transfer Agent (MTA – I learned a new acronym!) in Ubuntu is Postfix. I started reading their online documentation, and at first I found it quite inscrutable. I kind of panicked and ordered a book on Postfix. But after reading the docs some more, it is starting to make sense, and seems quite easy to configure it to just forward email. I’m glad I bought the book though. It hasn’t arrived yet, but I’m sure it will be handy down the road. I don’t currently have a need for locally hosted mailboxes, spam filters, etc. One thing I can definitely appreciate: administering mail for a large network is very complicated. My use case is pretty simple though, I hope.
I did some poking around on my Ubuntu equipped laptop. Configuring Apache should be easy. They made support for virtual hosts very easy right out of the box by how they arranged the configuration files.
So right now I am taking some notes about what to backup, making lists of databases, database users, collations, etc., all in preparation for RI-Day (Re-Imaging Day). I’ve got lots of stuff to juggle. I surprised myself by how much custom stuff I added in little over a year, and it will take some time to get it set back up on the new server. Therefore I am going to have to give myself a lot of time to do this. I’m thinking Memorial Day weekend would be a good target as I have some additional days off around then.
I think it will be a very good step to move to Ubuntu and I’m looking forward to it! I’m curious as to how my host has customized the image. Surely they have some things like DNS and SSH configured, otherwise no one could remotely login and administer it out of the box. But did they also setup mail? I guess I’ll find out.
25 Apr 09 Server Migration?
I’ve been renting a dedicated server from 1&1 Hosting (yes that’s my affiliate link) for a little over a year now. I’ve been very happy with it, it has allowed me to run several websites, host an IRC server, a TeamSpeak server, Subversion, and Trac. I even very briefly ran a Call of Duty server on it! It is currently running Fedora Core 6, which was old even when I got it a year ago. I’ve been itching to re-image it to something that is supported, and I think I’m finally going to do it. It makes sense to get this done before I go live with SG101 2.0. The only question now is what OS should I go with?
A year ago there wasn’t very many OS options with 1&1. Now, they have 17 variants, including CentOS, openSUSE, Debian, and Ubuntu, with various permutations of the Plesk control panel. I think the safe choice here is to switch to CentOS 5 with Plesk 9. This distribution is currently very similar to FC6 and comes with a newer version of Plesk, which I rely on a lot. However, it would be very cool to run Ubuntu 8.04 LTS, since that is also what I am developing with on my laptop. I think it would minimize surprises to have the same OS on both production and development. And, I’d really like to use at least Python 2.5. FC6 and CentOS 5 are still using Python 2.4.
Unfortunately, 1&1 does not offer Plesk with this configuration. Hmmm, can I take the training wheels off and manage a server without Plesk? Here are the things that Plesk makes easy for me with their web GUI:
- Creation of domains and subdomains.
- Managing the mail server.
- Rotating Apache server logs.
- Easy access to phpMyAdmin.
- Quick dashboard like status for how much memory and disk space I am using.
- Webstats.
- Maybe some things I don’t know about? This frightens me.
So the question becomes, can I manage these things without Plesk? Let’s see:
- Apache. I’ve had to heavily customize the Apache configuration files generated by Plesk in order to get Python apps running. I think I’m comfortable now with Apache, and I can just save off the Plesk configuration files for future reference. I don’t anticipate any problems with this.
- Mail. This is a big mystery to me. I didn’t even know the server was running QMail until last night. Fortunately, I don’t make heavy use of the mail server. I only use it for simple redirects. I have about 10 or so redirects. Getting the equivalent functionality running under Ubuntu may require some research on my part.
- Log rotation. I’m sure I could come up with some cron jobs to handle log rotation. I assume Plesk is using cron for this, and perhaps I can see how it is doing it.
- phpMyAdmin. I’m pretty sure I can just install phpMyAdmin myself using an Ubuntu package.
- Server stats. Not a huge loss, I can use the command line to get the same information.
- Webstats. I could probably install this myself, but to be honest, I don’t really look at these that much.
- The unknown. It should be “interesting” to find out what I am missing. Gulp.
So after thinking long and hard about this, I think going with Ubuntu will be a good choice for the long term. Having similar production and development environments will be important. Ubuntu 8.04 LTS still has (I think) about 4 years left for server support, and there should be a migration path to the next LTS release.
However leaving Plesk behind will require me to become more of a server geek. This, I don’t actually mind, as I am enjoying learning all this stuff slowly over time. I can practice some of these things on my laptop before I make the big switch. I need to figure out the mail server thing, and doing a dry run of moving the MySQL databases from production to my development server.
Another important question is when to do this? Ideally I will want to give myself plenty of time, perhaps over a long weekend. It will mean downtime for the current hosted websites. And it will probably be pretty nerve wracking. 
I’ll be thinking about these things and blogging about them over the course of the next few days.
