19 Sep 09 Server Monitoring with Munin
I finally got around to investigating server monitoring tools, and came across Munin. This tool allows you to monitor various nodes in your network, and it generates pretty graphs for a wide variety of system parameters. You install munin-node software on the machines you wish to monitor. The node software gathers statistics which are collected periodically by a munin master agent, which produces the nice graphs that you can view with a web browser.
Installing it was easy with Ubuntu, as packages already exist for it. It was a simple “sudo apt-get install munin munin-node.” The Ubuntu package set me up for running both the master and node software on the same machine. I then had to edit the /etc/munin/munin.conf to change a few paths around, notably where to put the generated HTML files. One hitch that stumped me for a few minutes was a permissions problem on the path to the location of the HTML files. The Ubuntu package creates a new munin user, so make sure all directories on the path to the HTML files have an “x” bit for world. Once that was sorted I started seeing data after 5 minutes for a variety of things including CPU load, MySQL statistics, and filesystem usage. Awesome! I decided to put Apache Basic Authorization password protection on the directory where the report files live, as it is sensitive data that I wouldn’t want just anyone to see.
Below is an example graph, one of about 50 (!) that are available right out of the box. Plugins exist and can be created for a wide variety of other parameters as well. In addition, Munin can notify you by email if parameters get out of range. See the Munin home page for more examples and information.
Munin MySQL Graph
Munin looks like a very useful tool to keep an eye on your server. Thanks to the Munin team for this great piece of software!
Tags: monitoring, munin, Server, tools
15 Sep 09 Django Tip: get_object_or_404() and select_related()
I was looking at the SQL my views were generating, and I came across a couple of places where I was using get_object_or_404(), and then later following some foreign keys in the returned object. Something like this:
forum = get_object_or_404(Forum, slug=slug) if not forum.category.can_access(request.user): return HttpResponseForbidden()
The problem is that two SQL queries occur here, one during the get_object_or_404(), and then another in the following if statement, when we access category, a foreign key on the forum object. It would sure be nice to somehow use a select_related() there to avoid the extra SQL query. I did some googling, and found a quick tip on one of the This Week in Django podcast pages. And yes, the documentation confirmed that get_object_or_404() can now take as a first argument either a model, a manager, or a queryset!
So now you can keep using the handy get_object_or_404() idiom, and reduce the number of queries with a slight bit of refactoring:
forum = get_object_or_404(Forum.objects.select_related(), slug=slug) if not forum.category.can_access(request.user): return HttpResponseForbidden()
Very cool!
04 Sep 09 Denyhosts: SSH Brute Force Protection
I’m always very uneasy when I look through my web server and system logs. It seems dozens of people (or more!) a month try to log into my server, attempt an SQL injection attack on one or more of my hosted websites, and probe for weaknesses. It really makes me lose some faith in humanity.
I was tipped of to the Denyhosts script via a tweet from @dozba. This is nothing more than a Python script that analyzes your ssh log file, looking for suspicious activity. It monitors the number of failed SSH login attempts over time, and once a configurable threshold is reached, it adds the IP address of the remote host to your hosts.deny file, thus blocking the brute-force break-in attempt. The script is very configurable. It can have different thresholds for login attempts to existent or non-existent user accounts, it can purge blocked entries, and it will email you when it blocks a host.
Installation and configuration on Ubuntu is a breeze, as it is available as a package. Use your favorite package manager and install it. Ubuntu automatically configures it to run in daemon mode, so you won’t need to mess with cron. Configure it by editing the file /etc/denyhosts.conf. Restart the daemon, and then you are done. In a nutshell:
$ sudo apt-get install denyhosts $ sudo vi /etc/denyhosts.conf $ sudo /etc/init.d/denyhosts restart
Two days after I installed it, I had already blocked 5 break-in attempts. Thanks to the author Phil Schwartz and the Ubuntu package maintainers for this very useful utility!
24 Jul 09 Call of Duty Random Map Rotation Generator Script
Here is a Python script I wrote to randomly generate a map rotation for our Call of Duty: World at War server. It outputs the “set sv_mapRotation” line that you should copy & paste into your server’s config file. Our gaming clan mainly plays TDM, but we got a second server to try out the other objective-based game types. Not knowing which map combinations were the best, I simply wrote a script to generate a random rotation of the non-TDM and non-DM gametypes. Unfortunately, I found out later that there is a 1024 character line length limit in the config file (boo!). So I later modified the script to observe this unfortunate limit. This means you can’t have very large map rotations.
Here are some notes about the script.
- The script should work with any Call of Duty game from 1 to W@W. You just have to edit the list of maps and use the appropriate game types. Just be advised that the game type names aren’t consistent across titles. For example, COD:MW uses “WAR” for team deathmatch, while the older games, and COD:W@W use the more familiar “TDM”. Not all game types are supported in all titles in the series, and some titles have game types that don’t exist in other games. E.g. Only COD1 has retrieval (“RE”) and behind enemy lines (“BEL”).
- Put the desired maps you want between the triple quoted lines 9 and 14, and separate them by spaces.
- The gametypes tuple on line 16 contains the game types supported. Edit this to contain the game types you want.
"""
map_rotate.py
Generate map rotation for COD:W@W.
25 June 2009 - Brian Neal
"""
import random
maps = """\
mp_airfield mp_asylum mp_castle mp_shrine mp_courtyard mp_dome mp_downfall
mp_hangar mp_kneedeep mp_makin mp_nachtfeuer mp_outskirts mp_roundhouse
mp_seelow mp_subway mp_suburban mp_makin_day
mp_docks mp_kwai mp_stalingrad
""".split()
gametypes = ('dom', 'koth', 'sab', 'sd', 'ctf', 'twar')
combos = []
for gametype in gametypes:
for map in maps:
combos.append((gametype, map))
random.shuffle(combos)
# There is a 1024 character limit for lines from the config file...damn.
s = 'set sv_mapRotation "'
for game in combos:
ss = 'gametype %s map %s ' % (game[0], game[1])
if len(s) + len(ss) > 1024:
break
s += ss
s = s.rstrip()
s += '"'
print s
Remember that the script is random, so you may get the same map back-to-back, and/or some maps may be missing due to the stupid 1024 character limit. I run it a few times, saving the result into a file, until I get one that looks good. If you are ambitious, you could modify it a bit to prevent these issues, although the code for doing something like that starts to get very complicated.
I hope some people find this useful!
13 Jun 09 Installing memcached for use with Python and Django
I installed memcached on my production server a while back. It’s supposed to be thee way to get fast and efficient caching for your Django powered website. I remember the process as being somewhat less than satisfying. Tonight I decided to get it running on my development box, which is running Ubuntu 8.04. So I took a lot of notes and present them here for my own future reference. I hope this may help someone. And as you can see, I have a few questions myself that perhaps someone can help me with. Read on…
(more…)
Tags: cmemcache, daemon, django, libmemcache, memcached
12 Jun 09 Blog Updated
I just upgraded to WordPress 2.8 using the automatic method they now provide. I have to say I was very impressed and happy that I didn’t have to break out the FTP client. You just type your FTP credentials into a form, hit submit, and it goes out and gets the latest WordPress tarball, unpacks it, and updates your database. Wow.
Of course it didn’t go off without a hitch. For some reason, any new directories that get created on my server have permissions 700 instead of the more usual 755. My webspace is owned by my user account, and Apache is considered “other” here, so it could not read the files in the new directories. Once I figured that out and did a chmod all was well.
The other drawback of using the automatic method is there may be a chance some php files are no longer being used anymore, and are gathering as cruft. I suppose I will do a clean manual install every once in a while to delete these orphan files.
12 Jun 09 Django and Python Logging
I started working on a simple Django application for accepting and recording Paypal donations. While I was working on the IPN code, it suddenly occurred to me that I really needed a way to log any errors that might occur. After all the IPN process will be initiated by Paypal completely out of my control (not counting the Paypal sandbox) and without any visual feedback. Thus, I’d like a record of the path through my code to make sure everything is working the way I expected.
I had learned about the brilliant Python logging module some time ago, and had even used it in my IRC bot application. But could you use this with Django?
I did some research and found a couple of blog posts and related Django projects. After studying them I came to the conclusion that indeed the Python logging module is an excellent way to add logging to your web application. And after working with it again, I am very impressed by the functionality that it offers and how easy it is to use. I would have killed to have something like this in my PHP days. So I laced some logging calls throughout my IPN listener code, and I’ll know soon enough if it works correctly. I’ll post more about this later. But for now, I’d like to add some links to the things I found useful related to logging in Django.
First there is this very useful blog post by Simon Willison titled “Debugging Django.” In addition to talking about logging, Simon also has tips on using the Python debugger, asserts, and some useful middleware.
Second, there is a Django application called django-logging. This application seems mainly aimed at getting your logging statements displayed at the bottom of your web pages while debugging a problem. Of course you could also hook in a logger to log to a file, which is more of what I wanted to do. Another useful feature of this application is that you can configure it to automatically log your application’s SQL queries.
And finally I looked at Fairview Computing’s Django request logging, or drlog. This project provides some middleware which adds the ability to add a unique identifier to each log entry to associate it with a given HTTP request. This allows you to easily trace a single request, even while multiple concurrent requests are happening.
Studying Simon’s blog post and the source code for the above two applications was very enlightening. In the end I decided to start with the bare Python logging facility for now, configuring it to write to a file. I was reassured to read that the Python logging module is thread safe. If I start using the logging module heavily, I may use the drlog middleware to help me map log statements to HTTP requests.
28 May 09 Server Migration: After Action Report
Well the dust has settled and everything has been migrated to the new server. The last big thing to move was surfguitar101.com. I started the DNS change Monday morning, and it was only a few hours later that it had propagated to my house, so I was able to switch on the new site. However, I made a typo in the ServerAlias line in the Apache configuration file, and the alias www.surfguitar101.com wasn’t working (I really wish people wouldn’t use that). I started hearing reports that people still couldn’t get into the site after 8 or 10 hours, but I just figured DNS hadn’t propagated to them yet. This seemed reasonable at the time, because irc.surfguitar101.com took nearly 20 hours to switch for me. But after a day and a half I checked my configuration, and sure enough, oops, a typo. A quick edit and a restart of Apache and everyone was good.
Things over all went very well. I’m just glad I several good books on Apache, DNS and bind, and the Postfix mail transfer agent. And again, it was a great decision to get a second server to migrate to instead of thinking I could just reimage the existing server and get everything back in a few hours.
Since I got everything running I’ve been tinkering around, trying to do a better job of admining this new server. I had a cron job that backed up my databases and SVN repository nightly, but the backups still got stored on the same machine. So I finally broke down and wrote a cool Python script that cron kicks off nightly to transfer the backups to the backup server my host provides. Python is so awesome. I used the Python ftplib to accomplish this. It’s a very useful library, but I thought the interface was kind of odd. You actually have to issue FTP protocol commands as strings, e.g. “STOR.” I think I would have tried to abstract the underlying protocol a bit, but I suppose you get maximum flexibility this way.
I also worked on understanding the System V style startup sequence a bit better. I have several programs that I want to execute as daemons whenever the server is restarted: IRC, my IRC bot, and Teamspeak to name a few. The builtin start-stop-daemon was very useful for this, as well as the script update-rc.d. For my IRC bot, which is written in Python, I found some great code to daemonize it here (thanks Sander!).
And finally, now that I have 3 virtual domains running on the server, I got concerned about all the Apache log files. Each website has both an access log and an error log. On the old server, Plesk had some scheme to rotate these logs. I’m on my own now. But some simple research revealed the already builtin logrotate program. Wow is this useful. I created a simple configuration file and now my logs are getting rotated and compressed weekly. I won’t have to worry about running out of disk space (under normal load anyway).
So with the server situation largely under control, I can finally start working on Django again and getting a beta test site going. More to come.
24 May 09 Changing Server Timezone in Ubuntu
I thought I would jot this down. To change the timzone of your Ubuntu server execute the following command:
$ sudo dpkg-reconfigure tzdata
This Ubuntu community wiki page has some good info on time in Ubuntu.
20 May 09 Server Upgrade Status: Getting There
Welcome, this is my first post from the new server!
I’m really, really glad I went with renting a 2nd server and migrating to it instead of my original plan of just re-imaging. I would have been down for days while I figured out Ubuntu without Plesk.
I got my new server and pressed the magic button to have it re-imaged to Ubuntu 8.04. It all seemed to work just fine. I spent last Saturday going to town on it, getting the version of Subversion I wanted to use built, getting mod_wsgi built, and other simple “new server” type tasks. But then I noticed that the following on Sunday:
/dev/md6 4.7G 228M 4.5G 5% /var /dev/md7 221G 83M 221G 1% /home
The sizes of /var and /home were swapped compared to the old server. I had a huge /var and a small /home before. I became very worried because on the old server, all the databases, website data, and logs all go under /var. Was I going to run out of room very quickly?
I stewed on this for a day or so, and I even toyed with the idea of reimaging to CentOS 5 and getting Plesk back. After all, my main goal was to get Python 2.5, and I subsequently learned you can run different versions of Python side by side. So in theory I could install Python 2.5 on CentOS if I built from source. Then I could tell mod_wsgi to use Python 2.5, meanwhile the rest of the system would still use Python 2.4, and things like the yum package manager would keep working.
But ultimately I decided I liked Ubuntu better. I liked the way they setup Apache for easy multiple virtual hosts. I liked the aptitude program as a simple GUI for apt-get. And Ubuntu is my development environment on my laptop, so having them both match would be a good thing.
I did ask my host if they could either customize the image to make /var bigger or help me do it. They said no to both. This wasn’t totally surprising or unexpected.
I decided it didn’t matter how big /var was. I’ll just put everything on /home, under a directory of my own making. I configured MySQL to use the new directory on /home, and all my virtual host websites will use it as well. Now only system logs will use /var, and it should be big enough for them.
With that settled I set of to try to make it work. I began moving deathofagremmie.com to the new server. Moving the database and WordPress files was easy enough. I then hacked my way through the DNS configuration, using the old server as a guide. I really wasn’t sure I had done it right, but what the heck, I thought I’d give it a try. I then had my host transfer the domain to the new server. There wasn’t much to do except wait for the DNS to propogate. I did this late at night, so I went to bed. The next day at work I noticed that it had transferred, and the website even came up! A few things didn’t work, but nothing I couldn’t fix at home later. But when I came home from work, DNS was still pointing to the old server. I had to wait until almost 24 hours had elapsed before it finally updated for me at home.
Once DNS was correct, I could more easily fix the problems from the migration. It turns out they were minor. I had copy/pasted some Apache configuration from the default files, and I had the “AllowOverride” option set to “none” on my virtual host. This meant Apache would not look for .htaccess files, and thus the mod_rewrite rules that WordPress uses weren’t working, and the authorization I had setup on the admin login was busted. No problem, I’d actually rather not have Apache look for .htaccess files for efficiency reasons. I just moved those directives out of .htacess and put them into my configuration file for the virtual host. Why didn’t I think of that before? But that wasn’t quite the whole story. Mod_rewrite still didn’t work, but that was only because it wasn’t enabled by default in Ubuntu. The Ubuntu team did a really good job of paring down the Apache configuration, and making it very easy to enable and disable modules. A simple “a2enmod” command and a graceful restart fixed it. The Ubuntu Server Guide was a great help to me while I was doing all this.
And yesterday I received my copy of Pro DNS and Bind by Ron Aitchison. By quickly reading the first few chapters the mystery around those DNS files was suddenly solved. I went back and redid those files with my new knowledge and I feel confident about them now. I highly recommend this book.
And, using my new copy of The Book of Postfix, I was able to get my mail aliases for deathofagremmie.com to work. Postfix was much easier to setup than I thought it would be. The developers did a great job of “just making it work” out of the box with minimal configuration. I don’t have any plans to have any real mailboxes on the server, I just want aliases to forward to my Google mail account. This was easy to setup, and is described in Chapter 14 “A Mail Server For Multiple Domains.” I used the virtual alias technique to accomplish this. Edit a few files, run some commands on them, restart and it all worked.
The last wrinkle of the night was figuring out how to change the hostname under Ubuntu. This was solved by doing some quick Googling. You basically have to edit /etc/hostname, /etc/hosts, then stop and start the hostname service. With that fixed, email from my new server looks correct.
So, what’s left? I’m liking this new configuration, so I think I’ll go with it. Now I have to move the two remaining sites: one is a Django powered site, and one is the current PHP based SurfGuitar101.com. After I get them moved and tested I can shut down the old server, and get back to working on the new Django powered version of SurfGuitar101.com.
