Since I was using my webcam from Windows XP, I decided to write my script under the new 1.7.1 version of Cygwin that came out recently. You’ll need to get both the Paramiko package and the python-crypto package.

Happy New Year everyone!

Things slowed down dramatically on SG101 2.0 this summer. There was the 2009 SG101 convention vacation and other fun summer things to distract me. I had put up a beta site for feedback but was still missing a forums application. I got the itch to start back up again sometime in the Fall. I decided to start on a forums application myself and to see how it went. If it was too complicated I would look for a third-party solution.

This is one case where I did look at several third-party applications for ideas and to check on their status. There doesn’t seem to be a single recognized forums application in Django-land. There are a number of them, and they range from very simple to moderately complex. Many of them seem unsupported and have obvious problems. So in the end, I decided that since the forums are probably the most important part of my site, it would be best if I wrote it myself so that I could understand it completely. This includes both strengths and weaknesses. I did borrow many ideas from existing applications, and some of my initial momentum came from djangobb. However, I quickly stopped looking at other apps because Django really makes it easy to write complex web applications once you get an idea and try a few things.

My forum app contains most of the functionality of the venerable phpBB-based board I have now. I added a few things like the ability for users to flag posts as spam or abuse (I sure wish I had that now). I am considering making the first few posts of a user require approval to counter spam. But I’m not sure it is worth the effort with the “flag post” feature in-place. I might just wait and see how well that works.

I also decided to save a user’s post read and unread status in the database, instead of using cookies. Too many of my existing users complain that when their cookies expire they lose track of which threads are new. It will cost some database space, no doubt, but it is an often requested feature to fix this issue. I implemented a rolling 7-day window of thread and post read status, and in initial tests it seems to work just fine. It did add significant complexity to the design however, and I’m not looking forward to debugging that logic when a problem occurs.

After finishing the forums, I began working on my lengthy to-do list using my Trac issue tracker. I also spent a great deal of time refactoring some of my original code that I wrote over a year ago. I’ve become so much more proficient with Python and jQuery it is inevitable. My task list has become quite small and I am thinking about wiping the existing beta site and putting up a new one over holiday break and launching an official beta test.

The one area that I am lacking in right now is a good design and layout. A few users have volunteered to help with that, and one in particular is showing me some really nice work. If I can just manage to implement his design we may be on to something. I may also try to reach out to someone who is familiar with Django.

There are a couple of interesting problems I either solved or worked around during this period that I should blog about. I’ll just have to find the time to do that. In particular, I wanted to share how I created an admin dashboard for user-created content that needs admin approval before being published.

I’ve also volunteered to give a “brown-bag” lunchtime talk at my employer on Python. I’ll have to prepare some slides over the holiday break for this.

So I’m not sure what shell=True does on Windows, and it certainly isn’t documented. I do not recommend using it unless you know what is happening  in the implementation. Interestingly, shell=False is the default. I should have left it that way.

Python leverages the GNU Readline Library in its shell code. The Readline library is used in a number of open source projects, notably the BASH shell. The Readline library looks for a configuration file at startup in a number of places. On Unix-like operating systems, the configuration file can be found at ~/.inputrc and falls back to /etc/inputrc.

One line is all you need in your ~/.inputrc file:

set editing-mode vi

Unfortunately I don’t know how to make this work under Windows.

Eventually as I got more and more excited about Python I decided to wholly embrace the customs of the community. But how could I easily tell my favorite editor Vim to use 4-space indents for just Python code? I still do a lot of C++ at work, and I still like 3-space indents in other file types.

My first attempt was to use a Vim modeline embedded right in my source files. At the bottom of my Python files I placed this line:

# vim: set sw=4 ts=4:

When Vim first opens a file, it scans the first and last few lines in the file, looking for modelines, and will use the settings just for that file.

This obviously worked, but soon I found it ugly and distracting to have that at the bottom of each file. And sometimes I would forget to put it there, and not realize it until a bunch of code had been written.

It turns out there is a far easier way, but it took some digging to discover. Vim has a very elaborate system of determing file types. If you peek into your Vim installation directory, you’ll see a ftplugin directory. This directory is full of little files that get executed once the type of a file has been determined. Some people on the Web suggest that you should modify these files directly. Thus I could modify the ftplugin/python.vim file, and place my tabstop and shiftwidth settings in there. But you would be modifying a Vim file, and your changes would get lost the next time you upgrade Vim.

There is a much better solution. After Vim executes the python.vim file in the ftplugins directory, it looks for a file called ~/.vim/after/ftplugin/python.vim (on Unix; on Windows I’ve had luck with C:\Progam Files\Vim\vimfiles\after\ftplugin\python.vim). If this file exists, it will be executed after the “official” python.vim file.

Inside your personal python.vim file, the documentation says to use the “setlocal” instead of “set” command. Thus, my file looks like:

setlocal tabstop=4
setlocal shiftwidth=4

Although to be honest, I’ve used the regular set for some time (by accident) without seeing any strange effects.

Happy Vimming!

I have used Vim on every programming task at every job I’ve had. I’m constantly learning new things about it, and the commands and keystrokes are now part of my DNA. I don’t even think how to execute commands anymore and I am often hard pressed to recall them to others learning Vim. I have to sit at a keyboard with Vim running in order to understand what my fingers are doing.

A while back, I discovered the auto-complete feature, where you type Ctrl-n in insert mode and Vim will attempt to complete the word you are typing based on all the words in all buffers and how frequently they occur. This is incredibly handy when programming to quickly pound out those function and variable names. I used “imap <S-Tab> <C-n>” in my .vimrc file to map this to shift-tab, which I felt was more convenient.

I also picked up a really handy way to cycle through all your buffers.  I used the tab and shift-tab mapping mentioned in that Vim tip. Combine that with Vim’s tab feature I can very easily find the file I am looking for.

And finally I had another big “wow, Vim can do that?” moment just last week. Since version 7, Vim has sported an “omni-complete” framework. Basically it is an architecture for people to hang code completion routines into Vim. In fact, right out of the box, Vim 7 has support for Javascript, CSS, and yes, even Python! Python completion is only available if you are using a version of Vim that has been compiled with Python support (e.g. on Ubuntu). Imagine my shock when I was editing a CSS file and I typed “display:” and then hit Ctrl-X Ctrl-O for the first time and a drop down list of all the possible values (inline, block, none, etc) appeared! No more hunting for that w3schools website. And the Python support is just as handy. Type “os.path.” followed by Ctrl-X Ctrl-O and you can see all the entities and function in the os.path module. I had no idea!

So all of this reminds me it is time to once again donate to the Vim project. Thanks Bram for all your amazing work you do on this invaluable tool.

P.S. I have no interest in the great editor holy war. Use a tool that you find intuitive and useful and learn it, back to front. I’m sure there is nothing wrong with other popular editors. But for me, Vim is it. I can’t imagine how many hours of productivity it has given me over the years.

Installing it was easy with Ubuntu, as packages already exist for it. It was a simple “sudo apt-get install munin munin-node.” The Ubuntu package set me up for running both the master and node software on the same machine. I then had to edit the /etc/munin/munin.conf to change a few paths around, notably where to put the generated HTML files. One hitch that stumped me for a few minutes was a permissions problem on the path to the location of the HTML files. The Ubuntu package creates a new munin user, so make sure all directories on the path to the HTML files have an “x” bit for world. Once that was sorted I started seeing data after 5 minutes for a variety of things including CPU load, MySQL statistics, and filesystem usage. Awesome! I decided to put Apache Basic Authorization password protection on the directory where the report files live, as it is sensitive data that I wouldn’t want just anyone to see.

Below is an example graph, one of about 50 (!) that are available right out of the box. Plugins exist and can be created for a wide variety of other parameters as well. In addition, Munin can notify you by email if parameters get out of range. See the Munin home page for more examples and information.

Munin MySQL Graph

Munin looks like a very useful tool to keep an eye on your server. Thanks to the Munin team for this great piece of software!

forum = get_object_or_404(Forum, slug=slug)
if not forum.category.can_access(request.user):
     return HttpResponseForbidden()

The problem is that two SQL queries occur here, one during the get_object_or_404(), and then another in the following if statement, when we access category, a foreign key on the forum object. It would sure be nice to somehow use a select_related() there to avoid the extra SQL query. I did some googling, and found a quick tip on one of the This Week in Django podcast pages. And yes, the documentation confirmed that get_object_or_404() can now take as a first argument either a model, a manager, or a queryset!

So now you can keep using the handy get_object_or_404() idiom, and reduce the number of queries with a slight bit of refactoring:

forum = get_object_or_404(Forum.objects.select_related(), slug=slug)
if not forum.category.can_access(request.user):
     return HttpResponseForbidden()

Very cool!

I was tipped of to the Denyhosts script via a tweet from @dozba. This is nothing more than a Python script that analyzes your ssh log file, looking for suspicious activity. It monitors the number of failed SSH login attempts over time, and once a configurable threshold is reached, it adds the IP address of the remote host to your hosts.deny file, thus blocking the brute-force break-in attempt. The script is very configurable. It can have different thresholds for login attempts to existent or non-existent user accounts, it can purge blocked entries, and it will email you when it blocks a host.

Installation and configuration on Ubuntu is a breeze, as it is available as a package. Use your favorite package manager and install it. Ubuntu automatically configures it to run in daemon mode, so you won’t need to mess with cron. Configure it by editing the file /etc/denyhosts.conf. Restart the daemon, and then you are done. In a nutshell:

$ sudo apt-get install denyhosts
$ sudo vi /etc/denyhosts.conf
$ sudo /etc/init.d/denyhosts restart

Two days after I installed it, I had already blocked 5 break-in attempts. Thanks to the author Phil Schwartz and the Ubuntu package maintainers for this very useful utility!