13 Jul 10 Django Middleware Gotcha
Here’s a quick explanation of a gotcha I ran into while writing my own Django process_response middleware. I had written some middleware to perform a “who’s online” type function some time ago. The middleware checks to see if a cookie is present. If not, it updates the database with a timestamp for the user, and then sets a cookie with a lifespan of 10 minutes. In this way I only have to update the database with “last seen” information about every 10 minutes, per user. Because I need to set a cookie, this processing had to be done in a process_response middleware. I had tested and debugged this middleware and I thought everything was fine.
But then yesterday I was checking various links on my site, and one link in particular generated a traceback from my middleware: “AttributeError: ‘WSGIRequest’ object has no attribute ‘user’.” What? Why was this happening on this one particular view and only this view?
One of the first things I do in my process_response function is check to see if the user is authenticated:
if request.user.is_authenticated(): ...
After putting a breakpoint in this code with pdb, I observed that indeed the request object my middleware received had no “user” attribute. This had me stupified until I suddenly realized that the link I had clicked on was of the form /xxx/yyy instead of /xxx/yyy/ due to a typo in my template. I forgotten the trailing slash. Aha, this was a clue, but I still couldn’t piece it together, and I had to get some sleep. I don’t know about you, but I hate going to sleep when you have a nagging unsolved problem.
Tonight I tackled the problem again. I remembered I have the APPEND_SLASH setting in my settings.py file set to True. This normalizes all my URLs such that they should all end in a trailing slash. This is implemented itself via a piece of middleware, the so called “Common” middleware which is baked into Django itself. I knew that there was some interaction going on between the CommonMiddleware and my own middleware. After reviewing the middleware docs to refresh my memory on how all this works, I turned my attention to my MIDDLEWARE_CLASSES setting:
MIDDLEWARE_CLASSES = ( 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'debug_toolbar.middleware.DebugToolbarMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'gpp.core.middleware.InactiveUserMiddleware', 'gpp.core.middleware.WhosOnline', 'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware', )
Because of the order above, the first piece of middleware to process a request is the CommonMiddleware. When it observed the lack of a trailing slash, it returned a HttpResponseRedirect object to redirect the user to the correct URL with a trailing slash. This immediately killed the the process_request chain of middleware processing. Thus, the AuthenticationMiddleware never got to run, and this is the middleware that attaches the user object to the request. Furthermore, since the request processing ended, we now start the process_response processing. This works through the above middleware list backwards. So when my WhosOnline middleware gets called, it is suddenly presented with a request object that has no user attribute. And bam, I hit my bug.
To fix this problem, one could reorder the list of middleware. But finding a middleware ordering that will satisfy all of your installed middleware is a kind of a maddening process that requires flipping through many doc pages. It just appeared to be too brittle of a solution. I’ll probably add some middleware in the future and completely upset the apple cart again. In the end, I decided to just guard against the case of not having a user attribute on the request object and just bail out early. In this use case, it isn’t important to see every single response. I can simply make it up on the next one. So I modified my code to look like this:
def process_response(self, request, response):
"""
Keep track of who is online.
"""
# Note that some requests may not have a user attribute
# as these may have been redirected in the middleware chain before
# the auth middleware got a chance to run. If this is the case, just
# bail out. We also ignore AJAX requests.
if not hasattr(request, 'user') or request.is_ajax():
return response
if request.user.is_authenticated():
...
This was a difficult and unexpected bug. However it produced one of those rare, genuine “light bulb” moments as it forced me to take a hard look at Django’s middleware processing in detail and finally learn it front to back.
Tags: django, middleware
03 Apr 10 jQuery UI Autocomplete Widget & Caching
I upgraded to the awesome new 1.4 version of jQuery recently in preparation for deploying another beta version of my site. Unfortunately the jquery-autocomplete plugin by Jörn Zaefferer doesn’t work with this new version. However, the equally awesome jQuery user interface library, jQuery UI, now has a new autocomplete widget that is based upon Jörn’s design. The API isn’t quite the same, but after some reading you’ll find that most of the functionality is still there. However the new UI widget does not support caching requests, leaving that to the user to implement. This kind of surprised me, as I would think that would be a commonly requested feature. But perhaps the UI folks felt that everyone would have different caching requirements. There is one example on the UI site of how to cache, but curiously it only caches the last request, which doesn’t seem all that useful to me.
In any event, after reading the documentation, it wasn’t hard to implement caching for my purposes. The following code simply caches the last 16 requests. Once the cache is full, it is flushed completely and ready to be filled again. Simple, crude, but hopefully somewhat effective. I’m not a Javascript expert, but this will work for me. I hope someone else finds it useful and can use it as a starting point for their use-case.
$(function() {
var cache = {};
var cacheSize = 0;
$("#id_of_your_text_box").autocomplete({
delay: 400,
minLength: 2,
source: function(request, response) {
if (cache[request.term]) {
response(cache[request.term]);
return;
}
$.ajax({
url: "/your/url/goes/here/",
type: "GET",
data: {
q: request.term,
limit: 10
},
dataType: "json",
success: function(data, textStatus) {
if (cacheSize >= 16) {
cache = {};
cacheSize = 0;
}
cache[request.term] = data;
++cacheSize;
response(data);
},
error: function(xhr, textStatus, ex) {
alert('Oops, an error occurred. ' + xhr.statusText + ' - ' +
xhr.responseText);
}
});
}
});
});
The key to making this to work is to provide your own source function to the autocomplete widget. The widget will call this function when it needs data. Inside the function we simply check to see if request.term is already in our cache. If it is, then we simply return the cached result via the response callback function. If not, we have to make an ajax call to the server to retrieve the data. When the ajax call completes, we store the result in our cache and then return it to the widget.
Again, this code simply stores the last 16 requests. When we run out of room we simply empty the cache and start over. A more sophisticated algorithm could use a different strategy like removing only the first item put in the cache or even the “least recently used” cache item. Break out your college computer science textbook on caching strategies and go nuts here.
The other feature this code doesn’t do is provide matching capabilities. For example, if the user has typed “foo” in the input box, this code will not return results for “bar foo”. If you need this, I suggest looking at the caching example on the UI site, or even at Jörn Zaefferer’s source code.
Tags: autocomplete, javascript, jquery, jquery-ui, plugins
20 Mar 10 Get Ready for Django 1.2
Django 1.2 is coming out soon. I had been sitting on a trunk version since last October or November, and I finally decided to update to try out some of the new features. This blog post will summarize my experience and report on any gotchas I ran into.
So What Changed?
First of all, I have to mention the great new site Django Advent. Django Advent was created as a way to publicize the new features and exciting changes coming in version 1.2. I assume this site was inspired by The 14 Days of jQuery, which did a similar thing for the great Javascript library jQuery. If you haven’t already, please visit Django Advent to get a nice overview of the changes. I developed a quick “shopping list” of features I wanted to add to my site after perusing Django Advent.
A more obvious way of finding out what changed is to check Django’s fine development documentation. In particular, check out the Django 1.2 release notes. Note that since 1.2 isn’t out yet, these notes are likely to change, so check back on them from time to time until the final release. These notes don’t tell the complete story. You’ll likely also want to read the Django Deprecation Timeline. The Django documentation is really great for an open-source project (or any project for that matter). I highly recommend spending some time familiarizing yourself with all the information that is available in the docs.
Adventures in Upgrading
So with some trepidation and giddiness, I finally did the svn up command and pulled down a hot off-the-press copy of trunk. What happened next? My site under development still ran, and after clicking around randomly I found nothing obviously broken. Yes, I know I need to get some tests written for my site and applications to make this more scientific and repeatable.
Multi-DB Support
One of the first things I did was cut my settings.py over to the new settings format for configuring your databases. That’s right, databases, plural. Although I don’t have an immediate use-case for this feature, I can easily see it becoming useful in the future. In any event, I appreciate the work that went into this, and it should help Django get more accepted in the enterprise world. Please read the Django Advent article on this feature for more background.
Cached Template Loader
The second change I made was to try out the cached template loader. This loader caches the compiled templates in your site’s cache, and thus Django doesn’t need to go to the filesystem (often multiple times) on every request to fetch template files. Again, read the Django Advent article (and this one) for more explanation of this great new feature.
This was very easy to setup, however I totally derailed myself when doing so. When I reviewed my settings.py and began to cut it over to the cached template loader, I threw out the “app_directories” loader from my list of template loaders. I didn’t need that, I have all my templates under a common templates directory (with sub-directories under it for my apps). I then happily confirmed my templates were being cached and went on to another task.
It wasn’t until a few days later that I noticed my admin wasn’t working; it couldn’t find the login template. Huh? And gee, my Admin docs stopped working too. Well after some flailing about, I realized that indeed *I* don’t use the app_directories loader, but several applications I didn’t write do. In particular, I was reminded that yes, the Django admin is, in fact, an application. Ha-ha! Whoops. Okay, I put the app_directories loader back and all was well.
The cached template loader will be useful in production, but I think I’m going to have to turn it off in development. I noticed already that when I change a template and then hit reload on my browser, gee, my change isn’t seen. This is not going to be a problem since settings.py is just a Python file, I’ll just conditionally use the correct loader for my current environment.
New Messages System
Once again, I refer you to the corresponding Django Advent article for an overview. A new messaging system has been put into place, replacing the old functionality that was tied to contrib.auth. I liked how this new system took the lead from the Python logging module for its design. I can imagine situations where it may be useful to filter messages at certain levels, for example.
It was very straight-forward to cut over to this new scheme. I appreciated being able to use the tags feature to tack on CSS styling to messages.
Syndication Changes
Many improvements were made to the syndication feed application. In particular, I liked the increased flexibility in the URL routing. It was easy to cut my syndication feed classes over to the new system, and along the way it looks like I was able to gain some additional RSS functionality thanks to the improvements in the base class.
I did run into one snag that was quickly resolved. I was using the cache_page decorator in my URLconf to cache the output of my feed classes. This stopped working after the upgrade. After I reported this problem on the Django Users mailing list, Django core developer Russell Keith-Magee confirmed it was a problem and wrote a ticket on this issue. Within hours it was resolved. Thanks Russell! Someday I hope to understand the root problem, which apparently is a bug in Python itself. I still need to do some more homework on how Python decorators work.
CSRF Updates
And finally, major updates to the CSRF protection system landed. Since I was not using this feature, I skipped over reading the release notes about it. Thus I was surprised when my login stopped working and started throwing CSRF related errors. It turns out that even though I am not using the CSRF middleware, all of the contrib applications, including admin and auth, have been cut over to use it. Normally this is not a problem, but as the upgrade notes state, if you aren’t using the provided contrib templates and you POST to a contrib view, things will stop working. The solution is to add the {% csrf_token %} to your custom template.
I am probably going to spend some time and cut all my applications over to use the CSRF protection. Django makes it easy to do, so it is really a no-brainer to add a bit of security to my site. It will be tedious to find all the existing forms in my templates to add the {% csrf_token %}, but that is a one-time task. I can easily add them to future forms as they are created.
The End?
Well that is all the issues I’ve run into so far, and as you can see, they are pretty minor and/or self-inflicted. But I hope this write-up will help other people on the fence about upgrading, or to just give them pointers on where to find upgrade information. Again, the Django Advent site combined with Django’s comprehensive documentation makes this upgrade easy. The new features rock and I can’t wait to incorporate them into my site and get some mileage on them. Thanks to all the Django developers and contributors for such a great piece of software.
22 Jan 10 Testing Your First Django App
Dougal Matthews wrote a great blog post entitled “Testing Your First Django App“. This is something that I have been meaning to do for a long time now, but didn’t know how to get started. Much to my surprise (I guess I shouldn’t be), Python supports the xUnit style of testing via the standard library package unittest. Since we are now using CxxTest at work, this is quite familiar to me now. Dougal’s blog entry shows some nice ways of testing Django web applications without using a server by mocking up requests and examining responses from your view functions. Very cool!
31 Dec 09 Python and SSH
I was fooling around with my webcam, and I got to wondering if I could write a small Python script to upload a snapshot to my webserver periodically. Since I was planning on uploading a new snapshot every ten minutes or so, I’d rather use SFTP to avoid sending my login and password unencrypted so frequently. I had already used FTP in Python before in a backup script, but I didn’t know if the Python standard library had a solution for SSH. After a bit of Googling, I discovered the 3rd party Python module Paramiko (the name is a combination of the Esperanto words for “paranoid” and “friend” — I love that), which allows you to do all kinds of SSH2 operations, including SFTP. Paramiko is extremely well documented (thank-you!) and I had no problems at all using it to accomplish my task. I also found this great introductory article by Python core developer Jesse Noller that was helpful in getting up to speed on it quickly.
Since I was using my webcam from Windows XP, I decided to write my script under the new 1.7.1 version of Cygwin that came out recently. You’ll need to get both the Paramiko package and the python-crypto package.
Happy New Year everyone!
20 Dec 09 SG101 2.0 Status Report
This is the obligatory “why haven’t you been blogging about your project in a while” post. Yes, I suppose it is time to give a quick update.
Things slowed down dramatically on SG101 2.0 this summer. There was the 2009 SG101 convention vacation and other fun summer things to distract me. I had put up a beta site for feedback but was still missing a forums application. I got the itch to start back up again sometime in the Fall. I decided to start on a forums application myself and to see how it went. If it was too complicated I would look for a third-party solution.
This is one case where I did look at several third-party applications for ideas and to check on their status. There doesn’t seem to be a single recognized forums application in Django-land. There are a number of them, and they range from very simple to moderately complex. Many of them seem unsupported and have obvious problems. So in the end, I decided that since the forums are probably the most important part of my site, it would be best if I wrote it myself so that I could understand it completely. This includes both strengths and weaknesses. I did borrow many ideas from existing applications, and some of my initial momentum came from djangobb. However, I quickly stopped looking at other apps because Django really makes it easy to write complex web applications once you get an idea and try a few things.
My forum app contains most of the functionality of the venerable phpBB-based board I have now. I added a few things like the ability for users to flag posts as spam or abuse (I sure wish I had that now). I am considering making the first few posts of a user require approval to counter spam. But I’m not sure it is worth the effort with the “flag post” feature in-place. I might just wait and see how well that works.
I also decided to save a user’s post read and unread status in the database, instead of using cookies. Too many of my existing users complain that when their cookies expire they lose track of which threads are new. It will cost some database space, no doubt, but it is an often requested feature to fix this issue. I implemented a rolling 7-day window of thread and post read status, and in initial tests it seems to work just fine. It did add significant complexity to the design however, and I’m not looking forward to debugging that logic when a problem occurs.
After finishing the forums, I began working on my lengthy to-do list using my Trac issue tracker. I also spent a great deal of time refactoring some of my original code that I wrote over a year ago. I’ve become so much more proficient with Python and jQuery it is inevitable. My task list has become quite small and I am thinking about wiping the existing beta site and putting up a new one over holiday break and launching an official beta test.
The one area that I am lacking in right now is a good design and layout. A few users have volunteered to help with that, and one in particular is showing me some really nice work. If I can just manage to implement his design we may be on to something. I may also try to reach out to someone who is familiar with Django.
There are a couple of interesting problems I either solved or worked around during this period that I should blog about. I’ll just have to find the time to do that. In particular, I wanted to share how I created an admin dashboard for user-created content that needs admin approval before being published.
I’ve also volunteered to give a “brown-bag” lunchtime talk at my employer on Python. I’ll have to prepare some slides over the holiday break for this.
13 Dec 09 Python Subprocess Popen and Windows
I was writing a Python script to automate some Subversion-related tasks at work last week on Windows. The Python code was calling the Subversion command-line client applications using the Python subprocess module. I don’t know why exactly, but I had passed in the parameter shell=True to the subprocess.Popen function. I probably did this because I saw a lot of examples on the web, and you know, monkey-see, monkey-do. In hindsight, these examples were probably for Unix. It turned out that this caused a lot of problems. The Subversion clients started complaining when handed filenames that contained “special” characters like ampersands. And when they got to the files that had both ampersands and spaces, all kinds of crazy errors were produced. I was pulling my hair out, trying to escape the special characters and/or use quotes around the filename, but I could not come up with a combination that would work for all the different filenames the script encountered. Finally, as I stared at the code, I somehow got the idea to try switching that shell=True to shell=False. Suddenly, everything worked perfectly. I undid all the escaping and quoting and everything worked as expected.
So I’m not sure what shell=True does on Windows, and it certainly isn’t documented. I do not recommend using it unless you know what is happening in the implementation. Interestingly, shell=False is the default. I should have left it that way. 
Tags: python, subprocess, svn, windows
30 Nov 09 Using the Python Shell in Vi Mode
The Python shell is an invaluable tool for trying things out and experimenting with the language. Wouldn’t it be great if you could use the Python shell in Vi mode? Much to my surprise, it turns out you can.
Python leverages the GNU Readline Library in its shell code. The Readline library is used in a number of open source projects, notably the BASH shell. The Readline library looks for a configuration file at startup in a number of places. On Unix-like operating systems, the configuration file can be found at ~/.inputrc and falls back to /etc/inputrc.
One line is all you need in your ~/.inputrc file:
set editing-mode vi
Unfortunately I don’t know how to make this work under Windows.
18 Nov 09 Vim: Different Settings For Different File Types
Old habits die hard. I have been using 3 space indents for C and C++ code for as long as I have been a programmer. A little over a year ago I dove head-first into Python, and eventually discovered PEP 8. PEP 8 is basically the style guide for the Python community. I read it with great interest and took just about everything to heart, except for the recommended 4-space indents. I just could not shake my old habit.
Eventually as I got more and more excited about Python I decided to wholly embrace the customs of the community. But how could I easily tell my favorite editor Vim to use 4-space indents for just Python code? I still do a lot of C++ at work, and I still like 3-space indents in other file types.
My first attempt was to use a Vim modeline embedded right in my source files. At the bottom of my Python files I placed this line:
# vim: set sw=4 ts=4:
When Vim first opens a file, it scans the first and last few lines in the file, looking for modelines, and will use the settings just for that file.
This obviously worked, but soon I found it ugly and distracting to have that at the bottom of each file. And sometimes I would forget to put it there, and not realize it until a bunch of code had been written.
It turns out there is a far easier way, but it took some digging to discover. Vim has a very elaborate system of determing file types. If you peek into your Vim installation directory, you’ll see a ftplugin directory. This directory is full of little files that get executed once the type of a file has been determined. Some people on the Web suggest that you should modify these files directly. Thus I could modify the ftplugin/python.vim file, and place my tabstop and shiftwidth settings in there. But you would be modifying a Vim file, and your changes would get lost the next time you upgrade Vim.
There is a much better solution. After Vim executes the python.vim file in the ftplugins directory, it looks for a file called ~/.vim/after/ftplugin/python.vim (on Unix; on Windows I’ve had luck with C:\Progam Files\Vim\vimfiles\after\ftplugin\python.vim). If this file exists, it will be executed after the “official” python.vim file.
Inside your personal python.vim file, the documentation says to use the “setlocal” instead of “set” command. Thus, my file looks like:
setlocal tabstop=4 setlocal shiftwidth=4
Although to be honest, I’ve used the regular set for some time (by accident) without seeing any strange effects.
Happy Vimming!
Tags: vim
24 Sep 09 I <3 Vim
I love the text editor Vim. I started using Vi because I really didn’t know any better around 1988 when I was first teaching myself C on my trusty Amiga 500. Not long after that I discovered Vim which sported many improvements over the venerable Vi. Sometime around 1993-1994 I started hacking Vim. I added a complete ARexx port to Vim so that you could script it. Alas, a new version of Vim came out right after I finished, and author Bram Moolenaar had changed the code so much I felt it was impossible to merge my changes into the new version. This was when I was a young pup programmer, and I was unfamiliar with revision control and difference/merging tools. I had fun doing it anyway, so I just chalked it up to experience.
I have used Vim on every programming task at every job I’ve had. I’m constantly learning new things about it, and the commands and keystrokes are now part of my DNA. I don’t even think how to execute commands anymore and I am often hard pressed to recall them to others learning Vim. I have to sit at a keyboard with Vim running in order to understand what my fingers are doing.
A while back, I discovered the auto-complete feature, where you type Ctrl-n in insert mode and Vim will attempt to complete the word you are typing based on all the words in all buffers and how frequently they occur. This is incredibly handy when programming to quickly pound out those function and variable names. I used “imap <S-Tab> <C-n>” in my .vimrc file to map this to shift-tab, which I felt was more convenient.
I also picked up a really handy way to cycle through all your buffers. I used the tab and shift-tab mapping mentioned in that Vim tip. Combine that with Vim’s tab feature I can very easily find the file I am looking for.
And finally I had another big “wow, Vim can do that?” moment just last week. Since version 7, Vim has sported an “omni-complete” framework. Basically it is an architecture for people to hang code completion routines into Vim. In fact, right out of the box, Vim 7 has support for Javascript, CSS, and yes, even Python! Python completion is only available if you are using a version of Vim that has been compiled with Python support (e.g. on Ubuntu). Imagine my shock when I was editing a CSS file and I typed “display:” and then hit Ctrl-X Ctrl-O for the first time and a drop down list of all the possible values (inline, block, none, etc) appeared! No more hunting for that w3schools website. And the Python support is just as handy. Type “os.path.” followed by Ctrl-X Ctrl-O and you can see all the entities and function in the os.path module. I had no idea!
So all of this reminds me it is time to once again donate to the Vim project. Thanks Bram for all your amazing work you do on this invaluable tool.
P.S. I have no interest in the great editor holy war. Use a tool that you find intuitive and useful and learn it, back to front. I’m sure there is nothing wrong with other popular editors. But for me, Vim is it. I can’t imagine how many hours of productivity it has given me over the years.
Tags: vim
