Fixed pages for Django

I have been using Django's flatpages app for some simple, static pages that were supposed to be temporary. I slapped a Javascript editor on the admin page and it has worked very well. However some of the pages have long outlived their "temporary" status, and I find myself needing to update them. It is then that I get angry at the Javascript editor, and there is no way to keep any kind of history on the page without having to fish through old database backups. I started to think it would be nice to write the content in a nice markup language, for example reStructuredText, which I could then commit to version control. I would just need a way to generate HTML from the source text to produce the flatpage content.

Of course I could use the template filters in django.contrib.markup. But turning markup into HTML at page request time can be more expensive than I like. Yes, I could cache the page, but I'd like the process to be more explicit.

In my first attempt at doing this, I wrote a custom management command that used a dictionary in my settings.py file to map reStructuredText files to flatpage URLs. My management command would open the input file, convert it to HTML, then find the FlatPage object associated with the URL. It would then update the object with the new HTML content and save it.

This worked okay, but in the end I decided that the pages I wanted to update were not temporary, quick & dirty pages, which is kind of how I view flatpages. So I decided to stop leaning on the flatpages app for these pages.

I then modified the management command to read a given input file, convert it to an HTML fragment, then save it in my templates directory. Thus, a file stored in my project directory as fixed/about.rst would get transformed to templates/fixed/about.html. Here is the source to the command which I saved as make_fixed_page.py:

import os.path
import glob

import docutils.core
from django.core.management.base import LabelCommand, CommandError
from django.conf import settings


class Command(LabelCommand):
    help = "Generate HTML from restructured text files"
    args = "<inputfile1> <inputfile2> ... | all"

    def handle_label(self, filename, **kwargs):
        """Process input file(s)"""

        if not hasattr(settings, 'PROJECT_PATH'):
            raise CommandError("Please add a PROJECT_PATH setting")

        self.src_dir = os.path.join(settings.PROJECT_PATH, 'fixed')
        self.dst_dir = os.path.join(settings.PROJECT_PATH, 'templates', 'fixed')

        if filename == 'all':
            files = glob.glob("%s%s*.rst" % (self.src_dir, os.path.sep))
            files = [os.path.basename(f) for f in files]
        else:
            files = [filename]

        for f in files:
            self.process_page(f)

    def process_page(self, filename):
        """Processes one fixed page"""

        # retrieve source text
        src_path = os.path.join(self.src_dir, filename)
        try:
            with open(src_path, 'r') as f:
                src_text = f.read()
        except IOError, ex:
            raise CommandError(str(ex))

        # transform text
        content = self.transform_input(src_text)

        # write output
        basename = os.path.splitext(os.path.basename(filename))[0]
        dst_path = os.path.join(self.dst_dir, '%s.html' % basename)

        try:
            with open(dst_path, 'w') as f:
                f.write(content.encode('utf-8'))
        except IOError, ex:
            raise CommandError(str(ex))

        prefix = os.path.commonprefix([src_path, dst_path])
        self.stdout.write("%s -> %s\n" % (filename, dst_path[len(prefix):]))

    def transform_input(self, src_text):
        """Transforms input restructured text to HTML"""

        return docutils.core.publish_parts(src_text, writer_name='html',
                settings_overrides={
                    'doctitle_xform': False,
                    'initial_header_level': 2,
                    })['html_body']

Next I would need a template that could render these fragments. I remembered that the Django include tag could take a variable as an argument. Thus I could create a single template that could render all of these "fixed" pages. Here is the template templates/fixed/base.html:

{% extends 'base.html' %}
{% block title %}{{ title }}{% endblock %}
{% block content %}
{% include content_template %}
{% endblock %}

I just need to pass in title and content_template context variables. The latter will control which HTML fragment I include.

I then turned to the view function which would render this template. I wanted to make this as generic and easy to do as possible. Since I was abandoning flatpages, I would need to wire these up in my urls.py. At first I didn't think I could use Django's new class-based generic views for this, but after some fiddling around, I came up with a very nice solution:

from django.views.generic import TemplateView

class FixedView(TemplateView):
    """
    For displaying our "fixed" views generated with the custom command
    make_fixed_page.

    """
    template_name = 'fixed/base.html'
    title = ''
    content_template = ''

    def get_context_data(self, **kwargs):
        context = super(FixedView, self).get_context_data(**kwargs)
        context['title'] = self.title
        context['content_template'] = self.content_template
        return context

This allowed me to do the following in my urls.py file:

urlpatterns = patterns('',
   # ...

   url(r'^about/$',
       FixedView.as_view(title='About', content_template='fixed/about.html'),
       name='about'),
   url(r'^colophon/$',
       FixedView.as_view(title='Colophon', content_template='fixed/colophon.html'),
       name='colophon'),

   # ...

Now I have a way to efficiently serve reStructuredText files as "fixed pages" that I can put under source code control.

Read and Post Comments

Upgrading to Django 1.4

Django 1.4 came out recently, and I took a few hours to upgrade my first site yesterday. I thought it would be useful for my own reference to write down what I did. I hope it will be useful to others. I'd love to read what you had to do, so if you went through this process and blogged about it, please leave a comment. Please keep in mind these aren't hard and fast steps or a recipe to follow, as my sites are probably nothing like yours and may use different features of Django.

Preparation

The first thing I did was to read very carefully the Django 1.4 release notes. The Django team does a great job of documenting what has changed, so it is well worth your time to read the release notes. It is also a good idea to at least skim the Django Deprecation Timeline. After reading these, you should make a list of the things you want to change, add, or remove.

Tips

After deciding what areas you want or need to change in your code, these tips may be useful to help you implement the changes.

  1. Run with warnings turned on. Use this command to run the development server: $ python -Wall manage.py runserver. Django makes use of Python's warning system to flag features that are deprecated. By running Python with the -Wall switch, you'll see these warnings in the development server output.
  2. Use the debugger to track down warnings. Not sure where a pesky warning is coming from? Just open the Django source code in your editor and put a import pdb; pdb.set_trace() line right above or below the warning. You can then use the debugger's w command to get a stack trace and find out exactly what code is leading to the warning. In my case I kept getting a few warnings with no idea where they were coming from. I used this technique to verify the warnings were coming from third party code and not my own. For more information on using the debugger (and you really should know how to use this invaluable tool), see the Pdb documentation.

My upgrade experience

Here is a list of things that I did during my port. Again, you may not need to do these, and the next site I upgrade may have a different list. All of these changes (except for the first) are described in the Django 1.4 release notes.

  1. Upgrade my Django debug toolbar. As of this writing, the Django debug toolbar I got from PyPI was not compatible with Django 1.4. I simply uninstalled it and grabbed the development version from GitHub with pip install git+https://github.com/django-debug-toolbar/django-debug-toolbar.git.
  2. Remove the ADMIN_MEDIA_PREFIX setting. The admin application in Django 1.4 now relies on the staticfiles application (introduced in Django 1.3) to handle the serving of static assets.
  3. Remove use of the {% admin_media_prefix %} template tag. Related to the above, this tag is now deprecated. I had a custom admin view that used this template tag, and I simply replaced it with {{ STATIC_URL }}/admin.
  4. Remove verify_exists on URLFields. The verify_exists option to the URLField has been removed for performance and security reasons. I had always set this to False; now I just had to remove it altogether.
  5. Add the require_debug_false filter to logging settings. As explained in the release notes, this change prevents admin error emails from being sent while in DEBUG mode.
  6. django.conf.urls.defaults is deprecated. I changed my imports in all urls.py files to use django.conf.urls instead of django.conf.urls.defaults to access include(), patterns(), and url(). The Django team had recently moved these functions and updated the docs and tutorial to stop using the frowned upon from django.conf.urls.defaults import *.
  7. Enable the new clickjacking protection. A nice new feature is some new middleware that adds the X-Frame-Options header to all response headers. This provides clickjacking protection in modern browsers.
  8. Add an admin password reset feature. By adding a few new lines to your urlconf you get a nifty new password reset feature for your admin.
  9. Update to the new manage.py. This was the biggest change with the most impact. The Django team has finally removed a long standing wart with its manage.py utility. Previously, manage.py used to play games with your PYTHONPATH which led to confusion when migrating to production. It could also lead to having your settings imported twice. See the next section in this blog entry for more on what I did here.

Reorganizing for the new manage.py

The change with the largest impact for me was reorganizing my directory structure for the new manage.py command. Before this change, I had organized my directory structure like this:

mysite/
   media/
   static/
   mysite/
      myapp1/
         __init__.py
         models.py
         views.py
         urls.py
      myapp2/
         __init__.py
         models.py
         views.py
         urls.py
      settings/
         __init__.py
         base.py
         local.py
         production.py
         test.py
      apache/
         myproject.wsgi
      logs/
      templates/
      manage.py
      urls.py
   LICENSE
   fabfile.py
   requirements.txt

After replacing the contents of my old manage.py with the new content, I then reorganized my directory structure to this:

mysite/
   media/
   static/
   myapp1/
      __init__.py
      models.py
      views.py
      urls.py
   myapp2/
      __init__.py
      models.py
      views.py
      urls.py
   myproject/
      settings/
         __init__.py
         base.py
         local.py
         production.py
         test.py
      apache/
         myproject.wsgi
      logs/
      templates/
      urls.py
   LICENSE
   fabfile.py
   manage.py
   requirements.txt

It is a subtle change, but I like it. It now makes it clear that my project is just an application itself, consisting of the top-level urls.py, settings, templates and logs. The manage.py file is now at the top level directory also, which seems right.

I had always made my imports as from app.models import MyModel instead of from myproject.app.models, so I didn't have to update any imports.

Since I use the "settings as a package" scheme, I did have to update the imports in my settings files. For example, in my local.py I had to change from settings.base import * to myproject.settings.base import *.

What I didn't do

Django 1.4's largest new feature is probably its support for timezones. I decided for this project not to take advantage of that. It would require a lot of changes, and it isn't really worth it for this small site. I may use it on the next site I convert to Django 1.4, and I will definitely be using it on new projects.

Conclusion

The upgrade process went smoother and quicker than I thought thanks to the excellent release notes and the Django team's use of Python warnings to flag deprecated features.

Read and Post Comments

Django Uploads and UnicodeEncodeError

Something strange happened that I wish to document in case it helps others. I had to reboot my Ubuntu server while troubleshooting a disk problem. After the reboot, I began receiving internal server errors whenever someone tried to view a certain forum thread on my Django powered website. After some detective work, I determined it was because a user that had posted in the thread had an avatar image whose filename contained non-ASCII characters. The image file had been there for months, and I still cannot explain why it just suddenly started happening.

The traceback I was getting ended with something like this:

File "/django/core/files/storage.py", line 159, in _open
return File(open(self.path(name), mode))

UnicodeEncodeError: 'ascii' codec can't encode characters in position 72-79: ordinal not in range(128)

So it appeared that the open() call was triggering the error. This led me on a twisty Google search which had many dead ends. Eventually I found a suitable explanation. Apparently, Linux filesystems don't enforce a particular Unicode encoding for filenames. Linux applications must decide how to interpret filenames all on their own. The Python OS library (on Linux) uses environment variables to determine what locale you are in, and this chooses the encoding for filenames. If these environment variables are not set, Python falls back to ASCII (by default), and hence the source of my UnicodeEncodeError.

So how do you tell a Python instance that is running under Apache / mod_wsgi about these environment variables? It turns out the answer is in the Django documentation, albeit in the mod_python integration section.

So, to fix the issue, I added the following lines to my /etc/apache2/envvars file:

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

Note that you must cold stop and re-start Apache for these changes to take effect. I got tripped up at first because I did an apache2ctrl graceful, and that was not sufficient to create a new environment.

Read and Post Comments