We want a Wiki!

Over at SurfGuitar101.com, we decided we'd like to have a wiki to capture community knowledge. I briefly looked at candidate wiki engines with an eye towards integrating them with Django, the framework that powers SurfGuitar101.com. And of course I was biased towards a wiki solution that was written in Python. I had tried a few wikis in the past, including the behemoth MediaWiki. MediaWiki is a very powerful piece of software, but it is also quite complex, and I didn't want to have to maintain a PHP infrastructure to run it.

Enter MoinMoin. This is a mature wiki platform that is actively maintained and written in Python. It is full featured but did not seem overly complex to me. It stores its pages in flat files, which seemed appealing for our likely small wiki needs. It turns out I had been a user of MoinMoin for many years without really knowing it. The Python.org wiki, Mercurial wiki, and omniORB wiki are all powered by MoinMoin. We'd certainly be in good company.

Single Sign-On

The feature that clinched it was MoinMoin's flexible authentication system. It would be very desirable if my users did not have to sign into Django and then sign in again to the wiki with possibly a different username. Managing two different password databases would be a real headache. The ideal solution would mean signing into Django would log the user into MoinMoin with the same username automatically.

MoinMoin supports this with their external cookie authentication mechanism. The details are provided in the previous link; basically a Django site needs to perform the following:

1. Set an external cookie for MoinMoin to read whenever a user logs into Django.
2. To prevent spoofing, the Django application should create a record that the cookie was created in some shared storage area accessible to MoinMoin. This allows MoinMoin to validate that the cookie is legitimate and not a fake.
3. When the user logs out of the Django application, it should delete this external cookie and the entry in the shared storage area.
4. Periodically the Django application should expire entires in the shared storage area for clean-up purposes. Otherwise this storage would grow and grow if users never logged out. Deleting entries older than the external cookie's age should suffice.

My Django Implementation

There are of course many ways to approach this problem. Here is what I came up with. I created a Django application called wiki to hold this integration code. There is quite a lot of code here, too much to conveniently show in this blog post. I will post snippets below, but you can refer to the complete code in my Bitbucket repository. You can also view online the wiki application in bitbucket for convenience.

Getting notified of when users log into or out of Django is made easy thanks to Django's login and logout signals. By creating a signal handler I can be notified when a user logs in or out. The signal handler code looks like this:

import logging
from django.contrib.auth.signals import user_logged_in, user_logged_out
from wiki.constants import SESSION_SET_MEMBER

logger = logging.getLogger(__name__)

def login_callback(sender, request, user, **kwargs):
    """Signal callback function for a user logging in.

    Sets a flag for the middleware to create an external cookie.

    """
    logger.info('User login: %s', user.username)

    request.wiki_set_cookie = True

def logout_callback(sender, request, user, **kwargs):
    """Signal callback function for a user logging in.

    Sets a flag for the middleware to delete the external cookie.

    Since the user is about to logout, her session will be wiped out after
    this function returns. This forces us to set an attribute on the request
    object so that the response middleware can delete the wiki's cookie.

    """
    if user:
        logger.info('User logout: %s', user.username)

        # Remember what Redis set member to delete by adding an attribute to the
        # request object:
        request.wiki_delete_cookie = request.session.get(SESSION_SET_MEMBER)


user_logged_in.connect(login_callback, dispatch_uid='wiki.signals.login')
user_logged_out.connect(logout_callback, dispatch_uid='wiki.signals.logout')

When a user logs in I want to create an external cookie for MoinMoin. But cookies can only be created on HttpResponse objects, and all we have access to here in the signal handler is the request object. The solution here is to set an attribute on the request object that a later piece of middleware will process. I at first resisted this approach, thinking it was kind of hacky. I initially decided to set a flag in the session, but then found out that in some cases the session is not always available. I then reviewed some of the Django supplied middleware classes and saw that they also set attributes on the request object, so this must be an acceptable practice.

My middleware looks like this.

class WikiMiddleware(object):
    """
    Check for flags on the request object to determine when to set or delete an
    external cookie for the wiki application. When creating a cookie, also
    set an entry in Redis that the wiki application can validate to prevent
    spoofing.

    """

    def process_response(self, request, response):

        if hasattr(request, 'wiki_set_cookie'):
            create_wiki_session(request, response)
        elif hasattr(request, 'wiki_delete_cookie'):
            destroy_wiki_session(request.wiki_delete_cookie, response)

        return response

The create_wiki_session() function creates the cookie for MoinMoin and stores a hash of the cookie in a shared storage area for MoinMoin to validate. In our case, Redis makes an excellent shared storage area. We create a sorted set in Redis to store our cookie hashes. The score for each hash is the timestamp of when the cookie was created. This allows us to easily delete expired cookies by score periodically.

def create_wiki_session(request, response):
    """Sets up the session for the external wiki application.

    Creates the external cookie for the Wiki.
    Updates the Redis set so the Wiki can verify the cookie.

    """
    now = datetime.datetime.utcnow()
    value = cookie_value(request.user, now)
    response.set_cookie(settings.WIKI_COOKIE_NAME,
            value=value,
            max_age=settings.WIKI_COOKIE_AGE,
            domain=settings.WIKI_COOKIE_DOMAIN)

    # Update a sorted set in Redis with a hash of our cookie and a score
    # of the current time as a timestamp. This allows us to delete old
    # entries by score periodically. To verify the cookie, the external wiki
    # application computes a hash of the cookie value and checks to see if
    # it is in our Redis set.

    h = hashlib.sha256()
    h.update(value)
    name = h.hexdigest()
    score = time.mktime(now.utctimetuple())
    conn = get_redis_connection()

    try:
        conn.zadd(settings.WIKI_REDIS_SET, score, name)
    except redis.RedisError:
        logger.error("Error adding wiki cookie key")

    # Store the set member name in the session so we can delete it when the
    # user logs out:
    request.session[SESSION_SET_MEMBER] = name

We store the name of the Redis set member in the user's session so we can delete it from Redis when the user logs out. During logout, this set member is retrieved from the session in the logout signal handler and stored on the request object. This is because the session will be destroyed after the logout signal handler runs and before the middleware can access it. The middleware can check for the existence of this attribute as its cue to delete the wiki session.

def destroy_wiki_session(set_member, response):
    """Destroys the session for the external wiki application.

    Delete the external cookie.
    Deletes the member from the Redis set as this entry is no longer valid.

    """
    response.delete_cookie(settings.WIKI_COOKIE_NAME,
                           domain=settings.WIKI_COOKIE_DOMAIN)

    if set_member:
        conn = get_redis_connection()
        try:
            conn.zrem(settings.WIKI_REDIS_SET, set_member)
        except redis.RedisError:
            logger.error("Error deleting wiki cookie set member")

As suggested in the MoinMoin external cookie documentation, I create a cookie whose value consists of the username, email address, and a key separated by the # character. The key is just a string of stuff that makes it difficult for a spoofer to recreate.

def cookie_value(user, now):
    """Creates the value for the external wiki cookie."""

    # The key part of the cookie is just a string that would make things
    # difficult for a spoofer; something that can't be easily made up:

    h = hashlib.sha256()
    h.update(user.username + user.email)
    h.update(now.isoformat())
    h.update(''.join(random.sample(string.printable, 64)))
    h.update(settings.SECRET_KEY)
    key = h.hexdigest()

    parts = (user.username, user.email, key)
    return '#'.join(parts)

Finally on the Django side we should periodically delete expired Redis set members in case users do not log out. Since I am using Celery with my Django application, I created a Celery task that runs periodically to delete old set members. This function is a bit longer than it probably needs to be, but I wanted to log how big this set is before and after we cull the expired entries.

@task
def expire_cookies():
    """
    Periodically run this task to remove expired cookies from the Redis set
    that is shared between this Django application & the MoinMoin wiki for
    authentication.

    """
    now = datetime.datetime.utcnow()
    cutoff = now - datetime.timedelta(seconds=settings.WIKI_COOKIE_AGE)
    min_score = time.mktime(cutoff.utctimetuple())

    conn = get_redis_connection()

    set_name = settings.WIKI_REDIS_SET
    try:
        count = conn.zcard(set_name)
    except redis.RedisError:
        logger.error("Error getting zcard")
        return

    try:
        removed = conn.zremrangebyscore(set_name, 0.0, min_score)
    except redis.RedisError:
        logger.error("Error removing by score")
        return

    total = count - removed
    logger.info("Expire wiki cookies: removed %d, total is now %d",
            removed, total)

MoinMoin Implementation

As described in the MoinMoin external cookie documentation, you have to configure MoinMoin to use your external cookie authentication mechanism. It is also nice to disable the ability for the MoinMoin user to change their username and email address since that is being managed by the Django application. These changes to the MoinMoin Config class are shown below.

class Config(multiconfig.DefaultConfig):

    # ...

    # Use ExternalCookie method for integration authentication with Django:
    auth = [ExternalCookie(autocreate=True)]

    # remove ability to change username & email, etc.
    user_form_disable = ['name', 'aliasname', 'email',]
    user_form_remove = ['password', 'password2', 'css_url', 'logout', 'create',
            'account_sendmail', 'jid']

Next we create an ExternalCookie class and associated helper functions to process the cookie and verify it in Redis. This code is shown in its entirety below. It is based off the example in the MoinMoin external cookie documentation, but uses Redis as the shared storage area.

import hashlib
import Cookie
import logging

from MoinMoin.auth import BaseAuth
from MoinMoin.user import User
import redis

COOKIE_NAME = 'YOUR_COOKIE_NAME_HERE'

# Redis connection and database settings
REDIS_HOST = 'localhost'
REDIS_PORT = 6379
REDIS_DB = 0

# The name of the set in Redis that holds cookie hashes
REDIS_SET = 'wiki_cookie_keys'

logger = logging.getLogger(__name__)


def get_cookie_value(cookie):
    """Returns the value of the Django cookie from the cookie.
    None is returned if the cookie is invalid or the value cannot be
    determined.

    This function works around an issue with different Python versions.
    In Python 2.5, if you construct a SimpleCookie with a dict, then
        type(cookie[key]) == unicode
    whereas in later versions of Python:
        type(cookie[key]) == Cookie.Morsel
    """
    if cookie:
        try:
            morsel = cookie[COOKIE_NAME]
        except KeyError:
            return None

        if isinstance(morsel, unicode):             # Python 2.5
            return morsel
        elif isinstance(morsel, Cookie.Morsel):     # Python 2.6+
            return morsel.value

    return None


def get_redis_connection(host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB):
    """
    Create and return a Redis connection using the supplied parameters.

    """
    return redis.StrictRedis(host=host, port=port, db=db)


def validate_cookie(value):
    """Determines if cookie was created by Django. Returns True on success,
    False on failure.

    Looks up the hash of the cookie value in Redis. If present, cookie
    is deemed legit.

    """
    h = hashlib.sha256()
    h.update(value)
    set_member = h.hexdigest()

    conn = get_redis_connection()
    success = False
    try:
        score = conn.zscore(REDIS_SET, set_member)
        success = score is not None
    except redis.RedisError:
        logger.error('Could not check Redis for ExternalCookie auth')

    return success


class ExternalCookie(BaseAuth):
    name = 'external_cookie'

    def __init__(self, autocreate=False):
        self.autocreate = autocreate
        BaseAuth.__init__(self)

    def request(self, request, user_obj, **kwargs):
        user = None
        try_next = True

        try:
            cookie = Cookie.SimpleCookie(request.cookies)
        except Cookie.CookieError:
            cookie = None

        val = get_cookie_value(cookie)
        if val:
            try:
                username, email, _ = val.split('#')
            except ValueError:
                return user, try_next

            if validate_cookie(val):
                user = User(request, name=username, auth_username=username,
                        auth_method=self.name)

                changed = False
                if email != user.email:
                    user.email = email
                    changed = True

                if user:
                    user.create_or_update(changed)
                if user and user.valid:
                    try_next = False

        return user, try_next

Conclusion

I've been running this setup for a month now and it is working great. My users and I are enjoying our shiny new MoinMoin wiki integrated with our Django powered community website. The single sign-on experience is quite seamless and eliminates the need for separate accounts.

The first nine

I had a lot of fun building Py-Enigma, an Enigma machine simulator library written in Python based on the information I found on Dirk's Technical Details of the Enigma Machine page. After I had built it and played with it for a while, I discovered Dirk also had an Enigma Challenge. I decided this would be a good test for Py-Enigma, and it may give me some ideas on how to improve the library for breaking messages.

The first nine messages were pretty easy to solve using Py-Enigma. Dirk gives you most of the key information for each problem, and one can then write a small program to brute force the missing information. I was able to solve the first nine messages in a weekend. This was very enjoyable, as it did require some thinking about how to simulate the problem and then executing it. I also learned about Python's itertools library, which contains some very handy functions for generating permutations and combinations for brute forcing a solution.

Dirk did a great job on the messages. They referenced actual events from World War II and I ended up spending a fair amount of time on Wikipedia reading about them. Translating the messages from German to English was a bit difficult, but I relied heavily on Google Translate.

Message 10

But then I came to the last message. Here, Dirk doesn't give you any key information, just ciphertext. Gulp. If you include 2 possible reflectors, the Enigma machine's key space for message 10 is somewhere around 2 x 10²³ keys, so it is impossible to brute force every possible key, unless you can afford to wait around for the heat death of the universe.

I then did some research, and learned about a technique for brute-forcing only the rotor and message settings, and then performing a heuristic technique called "hill-climbing" on the plugboard settings. I am deeply indepted to Frode Weierud and Geoff Sullivan, who have published high level descriptions of this technique. See Frode's The Enigma Collection page and Geoff's Crypto Barn for more information.

Python is too slow, or is it?

After a week or so of studying and tinkering, I came up with a hill-climbing algorithm in Python. I then started an attempt to break message 10, but after running the program for a little over 24 hours, I realized it was too slow. I did a "back of the envelope" calculation and determined I would need several hundred years to hill-climb every rotor and message setting. This was a bit disappointing, but I knew it was a possibility going in.

I then set out, somewhat reluctantly, to port Py-Enigma to C++. These days I don't do any C++ at home, only at work. However I got much happier when I decided this was an opportunity to try out some new C++11 features (the embedded compiler we use at work has no support for C++11). Okay, things are fun again! I have to say that C++11 is really quite enjoyable, and I made good use of the new auto style variable declarations, range-based for-loops, and brace-style initialization for containers like vector and map.

C++ is too slow, or is it?

It took a fair amount of time to re-tool my code to C++11. I ensured all my unit tests originally written in Python could pass in C++, and that I could solve some of the previous messages. I then re-implemented my hill-climbing algorithm in C++ and made another attempt at cracking message 10.

My C++ solution was indeed faster, but only by a factor of 8 or 9. I calculated I would need about 35 years to hill-climb all rotor and message settings, and I didn't really want to see if I could out-live my program. My morale was very low at this point. I wasn't sure if I could ever solve this self-imposed problem.

Algorithms & Data Structures are key

I had wanted to test myself with this problem and had avoided looking at anyone else's source code. However, at this juncture, I needed some help. I turned to Stephan Krah's M4 Message Breaking Project. Stephan had started a distributed computing attack on several 4-rotor naval Enigma machine messages. He had graciously made his source code available. Perhaps I could get some hints by looking at his code.

Indeed, Stephen's code provided the break I needed. I discovered a very cool trick that Stephen was doing right before he started his hill-climb. He pre-computed every path through the machine for every letter of the alphabet. Since hill-climbing involves running the message through the machine roughly 600 to 1000 times, this proved to be a huge time saver. After borrowing this idea, my hill-climbing times for one fixed rotor & message setting went down from 250 milliseconds to 2!

I immediately stopped looking at Stephen's code at this point; I didn't bother to compare our hill-climbing algorithms. I believe both of us are influenced by Weierud & Sullivan here. I was now convinced I had the speed-up I needed to make a serious attempt at message 10 again.

This showed me how critically important it is to have the right data structure and algorithm for a problem. However there were two other lessons I needed to learn before I was successful.

Utilize all your computing resources

Up to this point my cracking program was linear. It started at the first rotor and message setting, then hill-climbed and noted the score I got. Then it proceeded to the next message setting and hill-climbed, noting the score of that attempt, and so on. There were two problems with this approach.

I realized if I could do some sort of multi-processing, I could search the key space faster. I thought about forking off multiple processes or maybe using threads, but that was over-thinking it. In the end, I added command-line arguments to the program to tell it where in the key space to start looking. I could then simply run multiple instances of my program. My development laptop, which runs Ubuntu, is from 2008, and it has 2 cores in it. My desktop PC (for gaming), has 4 cores! I next spent a couple nights installing MinGW and getting my application to build in that environment. I could now run 6 instances of my cracking program. Hopefully this would pay off and shorten the time further.

(I didn't even attempt to commandeer my wife's and kids' computers. That wouldn't have gone down so well.)

How do you know when you are done?

The final problem that I had was determining when my cracking program had in fact cracked the message. The hill-climbing attempts produced scores for each key setting. Up to this point, based on running my program on messages 1 through 9, I had noticed that when I got a score divided by the message length of around 10 I had cracked the message. But my tests on messages 1 through 9 were hill-climbing using the known ring settings. My message cracker was using a shortcut: I was only searching ring settings on the "fast" rotor to save time. This would let me get the first part of a message, but if the middle or left-most rotor stepped at the wrong time the rest of the message would be unintelligible.

To verify my message cracking program, I ran it on some of the previous messages. I was shooting for a score divided by message length of 10. Using this heuristic, I was in fact able to crack some of the previous messages, but not others. It took me a while to realize that not searching the middle rotor's ring setting was causing this. The light bulb came on, and I realized that my heuristic of 10 is only valid when I search all the ring settings. Instead I should just keep track of the highest scores. When you get close enough, the score will be significantly higher than average score. Thus I may not know when I am done until I see a large spike in the score. I would then have to write another program to refine the solution to search for the middle and left-most ring setting.

Success at last

It took a little over a month of evenings and weekends to get to this point. I even had a false start where I ran my program on 6 cores for 1.5 days only to realize I had a bug in my code and I was only searching ring setting 0 on the fast ring. But once I had that worked out, I started a search on 6 cores on a Sunday night. I checked the logs off and on Monday, but no significant spike in the scores were noted. Before leaving for work on Tuesday I checked again, and noticed that one instance of my program had found a score twice as high as any other. Could that be it? I really wanted to stay and find out, but I had to go to work! Needless to say I was a bit distracted at work that day.

After work, I ran the settings through my Python application and my heart pounded as I recognized a few German words at the beginning of the message. In fact I had gotten around 75% of the message, then some garbage, but then the last few words were recognizable German. I then wrote another application to search the middle ring setting space. Using the highest score from that program allowed me to finally see the entire message. I cannot tell you how happy and relieved I was at the same time!

Final thoughts

For a while I didn't think I was going to be smart enough or just plain up to the task of solving message 10. Looking back on it, I see a nice progression of trying things, experimenting, setbacks, and a few "a-ha" moments that lead to the breakthrough. I am very grateful to Dirk Rijmenants for creating the challenge, which was a lot of fun. And I also wish to thank Frode Weierud and Geoff Sullivan for their hill-climbing algorithm description. Thanks again to Stephan Krah for the key speedup that made my attempt possible.

For now I have decided not to publish my cracking program, since it is a tiny bit specific to Dirk's challenge. I don't want to ruin his Enigma Challenge by giving anything away. I don't believe my broad descriptions here have revealed too much. Other people need to experience the challenge for themselves to fully appreciate the thrill and hard work that go into code breaking.

I should also note that there are other ways of solving Dirk's challenge. You could, with a bit of patience, solve messages 1 through 9 with one of the many Enigma Machine GUI simulators (Dirk has a fine one). I'm not certain how you could break message 10 using a GUI simulator and trial and error, however.

It is my understanding that this style of Enigma message breaking was not the technique used by the allied code-breakers during the war. I believe they used knowledge of certain key words, or "cribs", in messages received in weather reports and other regular messages to find the key for the day. I look forward to reading more about their efforts now.

Finally, I have published the C++11 port of Py-Enigma, which I am calling (rather unimaginatively) Cpp-Enigma on bitbucket. I hope Cpp-Enigma and Py-Enigma are useful for other people who want to explore the fascinating world of Enigma codes.

Py-Enigma

For some strange reason, I don't even remember why or how, I found myself browsing the Wikipedia page for the Enigma Machine. I quickly became fascinated and started surfing around some more. I found several Enigma machine simulators, some of them even online. It suddenly occured to me that it would be a fun project to try and write my own in Python. I wanted to challenge myself to see if I could figure it all without help, so I vowed not to look at anyone else's source code.

In order to write an Enigma simulator, you need a really good technical explanation of how it worked, along with numerous details like how the rotors were wired. Fortunately I very quickly found Dirk Rijmenants' incredible Cipher Machines and Cryptology website. In particular, his Technical Details of the Enigma Machine page was exactly what I was looking for, a real gold mine of information.

I had a long Memorial Day weekend, so I spent many hours in front of the computer, consuming every word of Dirk's explanations and trying to sort it all out in my head. And so a very fun marathon hacking session began. In the end I got it figured out, and you cannot believe how excited I was when I was able to decode actual Enigma messages from World War II!

And so Py-Enigma was born! Py-Enigma is a historically accurate simulation library for war-time Enigma machines, written in Python 3. I also included a simple command-line application for doing scripting or quick commad-line experiments.

Lessons learned

Since I didn't really know what I was doing at first, I wrote little classes for each of the components and unit tested them. I'm really glad I did this because not only did it find bugs, it also made me think harder about the problem and made me realize I didn't understand everything. When you make a mistake in cryptography, very often the answer you get is gibberish and there is no way to tell how close you are to the right answer. This was almost the case here, and I think I stumbled upon an Enigma weakness that the allied codebreakers must have seen also. I had a bug in my rotor stepping algorithm, and I got the right answer up until the point where the right rotor stepped the middle rotor, then the output went all garbage. Once I noticed this, I was able to focus on the stepping algorithm and find the bug. I'm sure the allied codebreakers must have experienced the same thing when they were guessing what rotors were being used for the message they were cracking.

I also decided to use this little project to really learn Sphinx. I had dabbled around in it before when I contributed some documentation patches to Django. I think writing the documentation took almost as long as my research and coding, but with Sphinx at least it was fun! It is a very useful and powerful package. I also checked out the awesome readthedocs.org and quickly got my documentation hosted there. What a fantastic service! Now whenever I push changes to bitbucket my docs are automatically built on readthedocs!

This was also my second project that I put up on PyPI. I'm a little bit more comfortable with the packaging process, but it is still a bit bewildering given all the choices.

Conclusion

I hope folks who are interested in history, cryptography, World War II, and Python find Py-Enigma useful and fun! You can get the code from either the Py-Enigma Bitbucket page or from PyPI. And a big thank-you to Dirk Rijmenants! Without his hard work and detailed explanation, Py-Enigma would have been considerably more difficult.

Preparation

The first thing I did was to read very carefully the Django 1.4 release notes. The Django team does a great job of documenting what has changed, so it is well worth your time to read the release notes. It is also a good idea to at least skim the Django Deprecation Timeline. After reading these, you should make a list of the things you want to change, add, or remove.

Tips

After deciding what areas you want or need to change in your code, these tips may be useful to help you implement the changes.

1. Run with warnings turned on. Use this command to run the development server: $ python -Wall manage.py runserver. Django makes use of Python's warning system to flag features that are deprecated. By running Python with the -Wall switch, you'll see these warnings in the development server output.
2. Use the debugger to track down warnings. Not sure where a pesky warning is coming from? Just open the Django source code in your editor and put a import pdb; pdb.set_trace() line right above or below the warning. You can then use the debugger's w command to get a stack trace and find out exactly what code is leading to the warning. In my case I kept getting a few warnings with no idea where they were coming from. I used this technique to verify the warnings were coming from third party code and not my own. For more information on using the debugger (and you really should know how to use this invaluable tool), see the Pdb documentation.

My upgrade experience

Here is a list of things that I did during my port. Again, you may not need to do these, and the next site I upgrade may have a different list. All of these changes (except for the first) are described in the Django 1.4 release notes.

1. Upgrade my Django debug toolbar. As of this writing, the Django debug toolbar I got from PyPI was not compatible with Django 1.4. I simply uninstalled it and grabbed the development version from GitHub with pip install git+https://github.com/django-debug-toolbar/django-debug-toolbar.git.
2. Remove the ADMIN_MEDIA_PREFIX setting. The admin application in Django 1.4 now relies on the staticfiles application (introduced in Django 1.3) to handle the serving of static assets.
3. Remove use of the {% admin_media_prefix %} template tag. Related to the above, this tag is now deprecated. I had a custom admin view that used this template tag, and I simply replaced it with {{ STATIC_URL }}/admin.
4. Remove verify_exists on URLFields. The verify_exists option to the URLField has been removed for performance and security reasons. I had always set this to False; now I just had to remove it altogether.
5. Add the require_debug_false filter to logging settings. As explained in the release notes, this change prevents admin error emails from being sent while in DEBUG mode.
6. django.conf.urls.defaults is deprecated. I changed my imports in all urls.py files to use django.conf.urls instead of django.conf.urls.defaults to access include(), patterns(), and url(). The Django team had recently moved these functions and updated the docs and tutorial to stop using the frowned upon from django.conf.urls.defaults import *.
7. Enable the new clickjacking protection. A nice new feature is some new middleware that adds the X-Frame-Options header to all response headers. This provides clickjacking protection in modern browsers.
8. Add an admin password reset feature. By adding a few new lines to your urlconf you get a nifty new password reset feature for your admin.
9. Update to the new manage.py. This was the biggest change with the most impact. The Django team has finally removed a long standing wart with its manage.py utility. Previously, manage.py used to play games with your PYTHONPATH which led to confusion when migrating to production. It could also lead to having your settings imported twice. See the next section in this blog entry for more on what I did here.

Reorganizing for the new manage.py

The change with the largest impact for me was reorganizing my directory structure for the new manage.py command. Before this change, I had organized my directory structure like this:

mysite/
   media/
   static/
   mysite/
      myapp1/
         __init__.py
         models.py
         views.py
         urls.py
      myapp2/
         __init__.py
         models.py
         views.py
         urls.py
      settings/
         __init__.py
         base.py
         local.py
         production.py
         test.py
      apache/
         myproject.wsgi
      logs/
      templates/
      manage.py
      urls.py
   LICENSE
   fabfile.py
   requirements.txt

After replacing the contents of my old manage.py with the new content, I then reorganized my directory structure to this:

mysite/
   media/
   static/
   myapp1/
      __init__.py
      models.py
      views.py
      urls.py
   myapp2/
      __init__.py
      models.py
      views.py
      urls.py
   myproject/
      settings/
         __init__.py
         base.py
         local.py
         production.py
         test.py
      apache/
         myproject.wsgi
      logs/
      templates/
      urls.py
   LICENSE
   fabfile.py
   manage.py
   requirements.txt

It is a subtle change, but I like it. It now makes it clear that my project is just an application itself, consisting of the top-level urls.py, settings, templates and logs. The manage.py file is now at the top level directory also, which seems right.

I had always made my imports as from app.models import MyModel instead of from myproject.app.models, so I didn't have to update any imports.

Since I use the "settings as a package" scheme, I did have to update the imports in my settings files. For example, in my local.py I had to change from settings.base import * to myproject.settings.base import *.

What I didn't do

Django 1.4's largest new feature is probably its support for timezones. I decided for this project not to take advantage of that. It would require a lot of changes, and it isn't really worth it for this small site. I may use it on the next site I convert to Django 1.4, and I will definitely be using it on new projects.

Conclusion

The upgrade process went smoother and quicker than I thought thanks to the excellent release notes and the Django team's use of Python warnings to flag deprecated features.