22 Jan 10 Testing Your First Django App
Dougal Matthews wrote a great blog post entitled “Testing Your First Django App“. This is something that I have been meaning to do for a long time now, but didn’t know how to get started. Much to my surprise (I guess I shouldn’t be), Python supports the xUnit style of testing via the standard library package unittest. Since we are now using CxxTest at work, this is quite familiar to me now. Dougal’s blog entry shows some nice ways of testing Django web applications without using a server by mocking up requests and examining responses from your view functions. Very cool!
15 Sep 09 Django Tip: get_object_or_404() and select_related()
I was looking at the SQL my views were generating, and I came across a couple of places where I was using get_object_or_404(), and then later following some foreign keys in the returned object. Something like this:
forum = get_object_or_404(Forum, slug=slug) if not forum.category.can_access(request.user): return HttpResponseForbidden()
The problem is that two SQL queries occur here, one during the get_object_or_404(), and then another in the following if statement, when we access category, a foreign key on the forum object. It would sure be nice to somehow use a select_related() there to avoid the extra SQL query. I did some googling, and found a quick tip on one of the This Week in Django podcast pages. And yes, the documentation confirmed that get_object_or_404() can now take as a first argument either a model, a manager, or a queryset!
So now you can keep using the handy get_object_or_404() idiom, and reduce the number of queries with a slight bit of refactoring:
forum = get_object_or_404(Forum.objects.select_related(), slug=slug) if not forum.category.can_access(request.user): return HttpResponseForbidden()
Very cool!
13 Jun 09 Installing memcached for use with Python and Django
I installed memcached on my production server a while back. It’s supposed to be thee way to get fast and efficient caching for your Django powered website. I remember the process as being somewhat less than satisfying. Tonight I decided to get it running on my development box, which is running Ubuntu 8.04. So I took a lot of notes and present them here for my own future reference. I hope this may help someone. And as you can see, I have a few questions myself that perhaps someone can help me with. Read on…
(more…)
Tags: cmemcache, daemon, django, libmemcache, memcached
12 Jun 09 Django and Python Logging
I started working on a simple Django application for accepting and recording Paypal donations. While I was working on the IPN code, it suddenly occurred to me that I really needed a way to log any errors that might occur. After all the IPN process will be initiated by Paypal completely out of my control (not counting the Paypal sandbox) and without any visual feedback. Thus, I’d like a record of the path through my code to make sure everything is working the way I expected.
I had learned about the brilliant Python logging module some time ago, and had even used it in my IRC bot application. But could you use this with Django?
I did some research and found a couple of blog posts and related Django projects. After studying them I came to the conclusion that indeed the Python logging module is an excellent way to add logging to your web application. And after working with it again, I am very impressed by the functionality that it offers and how easy it is to use. I would have killed to have something like this in my PHP days. So I laced some logging calls throughout my IPN listener code, and I’ll know soon enough if it works correctly. I’ll post more about this later. But for now, I’d like to add some links to the things I found useful related to logging in Django.
First there is this very useful blog post by Simon Willison titled “Debugging Django.” In addition to talking about logging, Simon also has tips on using the Python debugger, asserts, and some useful middleware.
Second, there is a Django application called django-logging. This application seems mainly aimed at getting your logging statements displayed at the bottom of your web pages while debugging a problem. Of course you could also hook in a logger to log to a file, which is more of what I wanted to do. Another useful feature of this application is that you can configure it to automatically log your application’s SQL queries.
And finally I looked at Fairview Computing’s Django request logging, or drlog. This project provides some middleware which adds the ability to add a unique identifier to each log entry to associate it with a given HTTP request. This allows you to easily trace a single request, even while multiple concurrent requests are happening.
Studying Simon’s blog post and the source code for the above two applications was very enlightening. In the end I decided to start with the bare Python logging facility for now, configuring it to write to a file. I was reassured to read that the Python logging module is thread safe. If I start using the logging module heavily, I may use the drlog middleware to help me map log statements to HTTP requests.
03 May 09 Django-Elsewhere
I just got finished integrating Leah Culver’s django-elsewhere application. Django-elsewhere was formerly Django-PSN (Portable Social Networks) and was originally created for the now defunct social networking site Pownce. This nifty application allows your users to add an arbitrary number of social networks, websites, and instant messengers to their profile. The application even comes with many icons for widely known sites.
In my previous design I had just stuck a few fields in my user profile for websites and a few of the common instant messengers. This was limiting, and I had been thinking about expanding it to a more general solution when I stumbled across this application.
To integrate it with my site, I created a template tag to display a user’s “elsewhere” sites, and I made a view and template to allow a user to edit their sites. This code was based off the example view and template that came with the application. In general the django-elsewhere code quality is quite high. There are still a few print statements in the code base, but that’s all I can find fault with right now.
Thank you django-elsewhere team for the big time saver!
Tags: django, django-elsewhere, sg101
12 Apr 09 Using html5lib to Sanitize User Input
Based on this blog post by Django co-BDFL Jacob Kaplan-Moss, I wanted to try using html5lib to sanitize user input. I’m using Markdown on most of the site. But in one particular place (news items), I am (currently) allowing users to submit HTML news stories with the TinyMCE Javascript editor. This is mainly because my users like to copy and paste content from sites like MySpace, and TinyMCE might be easier for them to use than Markdown. I may revisit this decision, but for now we’ll go with it.
I was using the lxml sanitizer for this purpose. But because of the high praises html5lib received from Jacob, and from studying the source code to both, html5lib gives me greater confidence, even if it is an order of magnitude slower. But, it isn’t like this is going to get used more than a few times a day, so that isn’t a concern.
Never having used html5lib, or any other HTML/XML parser before, it was a bit confusing to figure out how to use it for this task. After studying the code and the html5lib news group, I came up with the following bit of code I thought I would share. Comments are extremely welcome.
import html5lib
from html5lib import sanitizer, treebuilders, treewalkers, serializer
def sanitizer_factory(*args, **kwargs):
san = sanitizer.HTMLSanitizer(*args, **kwargs)
# This isn't available yet
# san.strip_tokens = True
return san
def clean_html(buf):
"""Cleans HTML of dangerous tags and content."""
buf = buf.strip()
if not buf:
return buf
p = html5lib.HTMLParser(tree=treebuilders.getTreeBuilder("dom"),
tokenizer=sanitizer_factory)
dom_tree = p.parseFragment(buf)
walker = treewalkers.getTreeWalker("dom")
stream = walker(dom_tree)
s = serializer.htmlserializer.HTMLSerializer(
omit_optional_tags=False,
quote_attr_values=True)
return s.render(stream)
I haven’t tested it extensively yet, but it seems to do the trick. I understand a future version of html5lib will have an option to strip completely out offending tags. Right now they are simply rendered harmless and remain in the input (via < and >). This is fine, as I can see them in the admin as I review submitted stories.
Tags: django, html5lib, sanitization, sg101
07 Apr 09 I contributed to Django!
Check out this ticket! Granted, it is just a typo fix to a Python docstring, but you got to start somewhere. 
What a great feeling in any event.
I hope I can contribute more meaningful features and bug fixes in the future.
Tags: django
29 Mar 09 Event Calendar: Time Zone Picker and Updates
I ended up creating a time zone picker for the event calendar. I saw the idea on the web somewhere. The problem is that there are nearly 400 common time zones in the database. Since every time zone is named in the format “area/location”, I created an area select and a location select. That broke up the time zones nicely, although some of the areas still have far too many entries to be completely convenient. I wrote a short Python script that parsed the pytz common time zones and generated a Javascript object literal to contain the select menus contents. Here is a screen shot showing it in action:
When you select an area (the left-most) control, the location select fills with the appropriate options. When the form is submitted, some Javascript runs to take the two select values and puts them together and populates a hidden time zone input field with the result. So, in the example above, when the form is submitted, the hidden field receives “US/Pacific”. Likewise, when the form is displayed, the hidden field is parsed and the two select controls are set accordingly. This works pretty well, although I think I could have done a better job of modularizing this code in case I need to use it in another place on the site (such as in a user’s profile). I will definitely do this later.
I’ve decided to tackle recurring events later, as it seems a bit involved, and as I stated, very few events on the calendar need this capability. So with the time zone picker in place, and the corresponding code on the server side (thanks to pytz), I can now accurately add events to the event calendar without losing local time information.
I also sat down finally and converted The Madeira’s website from mod_python to mod_wsgi. This wouldn’t have been possible without the excellent documentation that mod_wsgi has. I feel this will scale better, and it will allow me to more easily run multiple Python web applications side by side. I am anxious to get a Trac issue tracker running as well as a beta version of the new site.
The rest of the weekend was spent working the “to-do” list for the site in preparation for deploying a beta version. I really do need to get an issue tracker going to capture all the ideas and work I need to complete.
15 Mar 09 Event Calendar: Everything But Repeating Events
Continuing along with the Google calendar integration, I built views to allow users to edit and delete the events they have previously submitted. These actions are simply requests; the admin still has to approve them. In the PHP version, these updates did not require admin approval. Here, they do, because the admin must supply the Google password to synchronize the events on the Google calendar. So the events simply get marked as “edit request” or “delete request”, which the admin must approve by toggling the status to “edit approved” or “delete approved”. The admin then visits the custom Google sync view, supplies the password, and the events are synchronized with Google via the gdata API. This seems to be working out well, although I’m not entirely happy with the workflow in the admin area to accomplish all this. Right now I have to visit the model change list, approve changes, then go to the custom Google sync view. We’ll see how this works out in practice.
I was very pleasantly surpised at how easy it was to add a link to my custom Google sync view in the admin area. I decided I wanted this link visible on all model views in the admin area (there is only one model right now: the event). I simply had to create this path in one of my template directories: admin/gcalendar/change_list.html (where gcalendar is the name of my application). Django will look to see if I have provided a change_list.html, and will use it in preference to the default one. And I didn’t even have to copy the whole template, I only needed to use template inheritance to override the block that provides the “object tools”. My change_list.html looks like this:
{% extends "admin/change_list.html" %}
{% block object-tools %}
{% if has_add_permission %}
<ul class="object-tools"><li><a href="add/{% if is_popup %}?_popup=1{% endif %}" class="addlink">Add {{ name }}</a></li>
<li><a href="google_sync/">Google Sync</a></li>
</ul>
{% endif %}
{% endblock %}
I just added the <li> and link to my “Google Sync” page. Here is a screen shot of my calendar event change list that shows this link in action. Check the upper right, next to the “Add” link. It was nice to see it was automatically styled to fit in with the rest of the admin.
Clicking on the link leads one to the custom admin view. Here, the list of approved events appears. The admin types in the Google password, and the view synchronizes (adds, updates, deletes) the events with Google. It might be a minor pain to enter your Google password every time, but I’d rather not hard-code or store the admin’s Google password in the database.
I think using Google calendar is going to be cool. I like the fact that people can add the events to their own calendars, invite other people, and set email or SMS text message reminders for these events. I also like that other people can embed the site calendar on their own websites or blogs to spread the word.
With that in place, all I have left to do is to support repeating events. And actually, as it stands right now, I can also handle multi-day events. This covers 99% of the events on SurfGuitar101.com already. There are only two weekly gigs on the calendar right now. I have worked on this off and on for 3 or 4 weeks now, and have replicated most of the PHP version’s functionality by leveraging Google and Django. It took me about 3 or 4 months before I had this much working when I initially developed the PHP-Nuke version.
Repeating events might be difficult, because the gdata API sort of punts on them. In order to specify a repeating event, I have to format a fairly complex string according to some iCalendar RFC. Yikes. I will now poke around and see if there are some Python libraries available that I could use for this, or perhaps it isn’t as bad as I think it is and I can do it by hand. Or perhaps I will leave repeating events for the future since they aren’t used that much. I am itching to get what I have created so far deployed so I can start beta-testing and getting feedback from users.
08 Mar 09 Event Calendar: Submitting Events
I’m continuing on in my experiment to leverage Google calendar for my site’s event calendar. I have built a form that allows users to submit new events subject to admin approval. I modeled the form after the GUI used by Google calendar. I also decided to used jQuery UI’s datepicker. jQuery UI just released version 1.7, which is compatible with jQuery 1.3. After reading this blog entry, I’ve decided to try letting Google host jQuery for me. Google is also hosting the standard jQuery UI themes CSS files, so this seems awfully convenient.
Check out the screenshot of the event submit form:
The datepicker is very useful, and will add a lot to the site, I think. My existing PHP application is using a much less elegant form. So far this form only handles non-repeating events. I will add support for those later.
I then turned my attention to the backend: how to synchronize my event models with Google calendar? It turns out there is already a Django application designed for this purpose: django-cal. It is designed in a very generic way and seems to be very well done. I decided I didn’t need its full functionality, but I did reference the source code when designing my back end. So a big thank-you to the creators of django-cal!
At this point I have enough functionality to add events and get them on the Google calendar. I have coded, but not tested, update and delete functionality. I am using the batch facility of the gdata API. In a typical work cycle, I need to approve anywhere from 1 to 7 or so events at once. This isn’t a lot of events, but I think doing it in a batch mode will speed a synchronization cycle up noticeably.
I store a “status” field in each event that keeps track of the event status: new, new and approved, edited, edited and approved, deleted, deleted and approved, and “on calendar”. When a user adds an event it becomes “new”. An admin reviews the event, and marks it “new and approved”. After synchronization, the status changes to “on calendar”. Likewise, my plan is to let users edit and request deletes for any events that they submit. My custom admin synchronization view will the perform the operations on all events with the “… and approved” status in a batch mode.
The custom admin view I wrote about in the last entry is also working well. I overrode the base django.contrib.admin template, so it looks well-integrated with the rest of the admin part of the site. I also figured out how to use the breadcrumbs for admin navigation, and noticed the base template will display a variable called “messages” with the nice green checkmark, if present. Likewise, the admin CSS has a class called “errorlist” for displaying errors. So it is quite easy to make a pretty professional looking custom admin screen by studying the base template and admin CSS files. I will post a screenshot of my custom admin view after I get some more events through the work cycle in a future blog post.
The Python gdata API isn’t as clumsy as I first thought it was. I was able to wrap it in a Calendar class to make it easier to use. If you need to update or delete an event in batch mode, it seems like you have to retrieve the event from Google first. I struggled against this initially, since I was storing away the “edit link” for each event I create. I was hoping to use the edit link to avoid another trip to Google for an update. But after searching the Google Calendar Data API group archives, it seems that is indeed recommended to retrieve any event prior to an update or delete, at least in batch mode.
This seems to be working out well so far. After I code and test the other operations I will blog with more details.
