msgbartop
by Brian Neal
msgbarbottom

03 May 09 Django-Elsewhere

I just got finished integrating Leah Culver’s django-elsewhere application. Django-elsewhere was formerly Django-PSN (Portable Social Networks) and was originally created for the now defunct social networking site Pownce. This nifty application allows your users to add an arbitrary number of social networks, websites, and instant messengers to their profile. The application even comes with many icons for widely known sites.

In my previous design I had just stuck a few fields in my user profile for websites and a few of the common instant messengers. This was limiting, and  I had been thinking about expanding it to a more general solution when I stumbled across this application.

To integrate it with my site, I created a template tag to display a user’s “elsewhere” sites, and I made a view and template to allow a user to edit their sites. This code was based off the example view and template that came with the application. In general the django-elsewhere code quality is quite high. There are still a few print statements in the code base, but that’s all I can find fault with right now.

Thank you django-elsewhere team for the big time saver!

Tags: , ,

12 Apr 09 Using html5lib to Sanitize User Input

Based on this blog post by Django co-BDFL Jacob Kaplan-Moss, I wanted to try using html5lib to sanitize user input. I’m using Markdown on most of the site. But in one particular place (news items), I am (currently) allowing users to submit HTML news stories with the TinyMCE Javascript editor. This is mainly because my users like to copy and paste content from sites like MySpace, and TinyMCE might be easier for them to use than Markdown. I may revisit this decision, but for now we’ll go with it.

I was using the lxml sanitizer for this purpose. But because of the high praises html5lib received from Jacob, and from studying the source code to both, html5lib gives me greater confidence, even if it is an order of magnitude slower. But, it isn’t like this is going to get used more than a few times a day, so that isn’t a concern.

Never having used html5lib, or any other HTML/XML parser before, it was a bit confusing to figure out how to use it for this task. After studying the code and the html5lib news group, I came up with the following bit of code I thought I would share. Comments are extremely welcome.

import html5lib
from html5lib import sanitizer, treebuilders, treewalkers, serializer

def sanitizer_factory(*args, **kwargs):
    san = sanitizer.HTMLSanitizer(*args, **kwargs)
    # This isn't available yet
    # san.strip_tokens = True
    return san

def clean_html(buf):
    """Cleans HTML of dangerous tags and content."""
    buf = buf.strip()
    if not buf:
        return buf

    p = html5lib.HTMLParser(tree=treebuilders.getTreeBuilder("dom"),
            tokenizer=sanitizer_factory)
    dom_tree = p.parseFragment(buf)

    walker = treewalkers.getTreeWalker("dom")
    stream = walker(dom_tree)

    s = serializer.htmlserializer.HTMLSerializer(
            omit_optional_tags=False,
            quote_attr_values=True)
    return s.render(stream)

I haven’t tested it extensively yet, but it seems to do the trick. I understand a future version of html5lib will have an option to strip completely out offending tags. Right now they are simply rendered harmless and remain in the input (via < and >). This is fine, as I can see them in the admin as I review submitted stories.

Tags: , , ,

29 Mar 09 Event Calendar: Time Zone Picker and Updates

I ended up creating a time zone picker for the event calendar. I saw the idea on the web somewhere. The problem is that there are nearly 400 common time zones in the database. Since every time zone is named in the format “area/location”, I created an area select and a location select. That broke up the time zones nicely, although some of the areas still have far too many entries to be completely convenient. I wrote a short Python script that parsed the pytz common time zones and generated a Javascript object literal to contain the select menus contents. Here is a screen shot showing it in action:

Time Zone Picker

When you select an area (the left-most) control, the location select fills with the appropriate options. When the form is submitted, some Javascript runs to take the two select values and puts them together and populates a hidden time zone input field with the result. So, in the example above, when the form is submitted, the hidden field receives “US/Pacific”. Likewise, when the form is displayed, the hidden field is parsed and the two select controls are set accordingly. This works pretty well, although I think I could have done a better job of modularizing this code in case I need to use it in another place on the site (such as in a user’s profile). I will definitely do this later.

I’ve decided to tackle recurring events later, as it seems a bit involved, and as I stated, very few events on the calendar need this capability. So with the time zone picker in place, and the corresponding code on the server side (thanks to pytz), I can now accurately add events to the event calendar without losing local time information.

I also sat down finally and converted The Madeira’s website from mod_python to mod_wsgi. This wouldn’t have been possible without the excellent documentation that mod_wsgi has. I feel this will scale better, and it will allow me to more easily run multiple Python web applications side by side. I am anxious to get a Trac issue tracker running as well as a beta version of the new site.

The rest of the weekend was spent working the “to-do” list for the site in preparation for deploying a beta version. I really do need to get an issue tracker going to capture all the ideas and work I need to complete.

Tags: , , , , ,

18 Mar 09 Event Calendar: Oh yeah, time zones…

Very shortly after I wrote the last blog entry I began having some nagging doubts about time zones. The current PHP version of the calendar is really time zone agnostic. It is assumed that when you see an event for California, for example, that you understand the time for the event is local to the Pacific time zone. I was kind of hoping to dodge this problem, but it seems unavoidable now that I am using Google calendar for the back-end.

The Google calendar has a time zone associated with it that I set when I created it. I am using my own local time zone: Central Standard Time, aka CST, aka GMT-6. Well, currently we are in daylight savings time (ack), so it is actually CDT, or GMT-5. When I initially added events to Google calendar, I naively just added them without providing any time zone information. Being somewhat new to Python, I haven’t totally come to grips with how Python deals with time zones, and I was just hacking with my blinders on. Well, needless to say, this didn’t work well, as Google simply assumed I was providing UTC times. Thus when they were displayed on my Google calendar, they had a nice 5 hour offset. Oops. Okay, to “fix” that, I peeked at django-cal, and noticed it was using a Django utility to provide the required time zone information. Using that technique, I got my events to show up in the correct local time on my Google calendar. Hooray!

But wait, that isn’t the whole story. Users can look at my Google calendar, and choose to copy events to their own Google calendars, which may have different time zones. Therefore, if a user submits an event to my calendar that takes place in California, my current code sets the time zone incorrectly. However this isn’t obvious as the time on my calendar will still be the same as the user entered it. But, if that user chooses to copy the event back to her California-based calendar, the event will suddenly be off by 2 hours. Crap!

So it looks like I do need to come to terms with time zones, and bite the bullet and ask the user what time zone the event is in. I am currently studying how Python handles time zones, and looking at pytz, a nice Python interface to the famous Olson database of timezones. Do I need to build some kind of time zone picker? I’m not finding a lot of those, which makes me wonder. In any event, this is seriously complicating my event calendar. But it is pretty interesting stuff and I hope I can get it right, as I really do want the events on my calendar to reflect accurate local times.

Tags: , , , ,

15 Mar 09 Event Calendar: Everything But Repeating Events

Continuing along with the Google calendar integration, I built views to allow users to edit and delete the events they have previously submitted. These actions are simply requests; the admin still has to approve them. In the PHP version, these updates did not require admin approval. Here, they do, because the admin must supply the Google password to synchronize the events on the Google calendar. So the events simply get marked as “edit request” or “delete request”, which the admin must approve by toggling the status to “edit approved” or “delete approved”. The admin then visits the custom Google sync view, supplies the password, and the events are synchronized with Google via the gdata API. This seems to be working out well, although I’m not entirely happy with the workflow in the admin area to accomplish all this. Right now I have to visit the model change list, approve changes, then go to the custom Google sync view. We’ll see how this works out in practice.

I was very pleasantly surpised at how easy it was to add a link to my custom Google sync view in the admin area. I decided I wanted this link visible on all model views in the admin area (there is only one model right now: the event). I simply had to create this path in one of my template directories: admin/gcalendar/change_list.html (where gcalendar is the name of my application). Django will look to see if I have provided a change_list.html, and will use it in preference to the default one. And I didn’t even have to copy the whole template, I only needed to use template inheritance to override the block that provides the “object tools”. My change_list.html looks like this:

{% extends "admin/change_list.html" %}
{% block object-tools %}
{% if has_add_permission %}
<ul class="object-tools"><li><a href="add/{% if is_popup %}?_popup=1{% endif %}" class="addlink">Add {{ name }}</a></li>
<li><a href="google_sync/">Google Sync</a></li>
</ul>
{% endif %}
{% endblock %}

I just added the <li> and link to my “Google Sync” page. Here is a screen shot of my calendar event change list that shows this link in action. Check the upper right, next to the “Add” link. It was nice to see it was automatically styled to fit in with the rest of the admin.

GCalendar Admin Events Change List

Clicking on the link leads one to the custom admin view. Here, the list of approved events appears. The admin types in the Google password, and the view synchronizes (adds, updates, deletes) the events with Google. It might be a minor pain to enter your Google password every time, but I’d rather not hard-code or store the admin’s Google password in the database.

GCalendar Admin Google Sync View

I think using Google calendar is going to be cool. I like the fact that people can add the events to their own calendars, invite other people, and set email or SMS text message reminders for these events. I also like that other people can embed the site calendar on their own websites or blogs to spread the word.

With that in place, all I have left to do is to support repeating events. And actually, as it stands right now, I can also handle multi-day events. This covers 99% of the events on SurfGuitar101.com already. There are only two weekly gigs on the calendar right now. I have worked on this off and on for 3 or 4 weeks now, and have replicated most of the PHP version’s functionality by leveraging Google and Django. It took me about 3 or 4 months before I had this much working when I initially developed the PHP-Nuke version.

Repeating events might be difficult, because the gdata API sort of punts on them. In order to specify a repeating event, I have to format a fairly complex string according to some iCalendar RFC. Yikes. I will now poke around and see if there are some Python libraries available that I could use for this, or perhaps it isn’t as bad as I think it is and I can do it by hand. Or perhaps I will leave repeating events for the future since they aren’t used that much. I am itching to get what I have created so far deployed so I can start beta-testing and getting feedback from users.

Tags: , , , ,

21 Feb 09 Event Calendar – Experiments with Google Calendar

I haven’t had a lot of time to explore implementing an event calendar with Google Calendar, but I did try a few things out. First of all, it is pretty easy to embed a Google Calendar into your website:

Google Calendar Embedded on Website

Google Calendar Embedded on Website

As I mentioned in the last post, I downloaded the Google Python gdata client library. I wrote a small Python script based on the provided example code. I was able to add events to my calendar from this Python script:

import gdata.calendar
import gdata.calendar.service
import gdata.service
import atom
import atom.service
import time

cal_client = gdata.calendar.service.CalendarService()
cal_client.email = 'xxx@gmail.com'
cal_client.password = 'xxx'
cal_client.source = 'Google-Calendar_Python_Sample-1.0'
cal_client.ProgrammaticLogin()

title = 'My Test Event from Python'
content = '''This is the content of the event in some fashion.
<a href="http://google.com">My Link</a>

http://surfguitar101.com.

<b>Bold</b> text.
<p>Here is a paragraph.</p>
<p>And another</p>
<ul><li>Item 1</li><li>Item 2</li></ul>
<img src="http://surfguitar101.com/modules/Forums/images/smiles/icon_cool.gif" alt="Smiley" />
'''
where = 'An undisclosed location.'
#cal_path = '/calendar/feeds/default/private/full'
cal_path = '/calendar/feeds/xxx@group.calendar.google.com/private/full'

event = gdata.calendar.CalendarEventEntry()
event.title = atom.Title(text=title)
event.content = atom.Content(text=content)
event.where.append(gdata.calendar.Where(value_string=where))

# Use current time for the start_time and have the event last 1 hour
start_time = time.strftime('%Y-%m-%dT%H:%M:%S.000Z', time.gmtime())
end_time = time.strftime('%Y-%m-%dT%H:%M:%S.000Z', time.gmtime(time.time() + 3600))
event.when.append(gdata.calendar.When(start_time=start_time, end_time=end_time))

new_event = cal_client.InsertEvent(event, cal_path)

As you can see, I was testing to see if I could insert HTML into the event, and if it would display correctly on the Google Calendar. According to my tests, it does appear you can insert simple HTML. The only thing that did not work was <img> tags. Unfortunately, if you log into your Google account, and then go to your Google Calendar and try to edit this event using the Google GUI, all HTML will be stripped. This will make it difficult to edit an existing event and will probably steer me towards keeping the user added event around in the database for future editing until that event has expired.

I have to say I am not really impressed with the Python API. It seems rather “un-Pythonic”. Why do I have to construct atom objects and assign them to the event title and content fields? The API should do this for me. Why do I have to format my own date strings? The API should accept Python datetime objects and do that for me. Pretty disappointing coming from Google.

I am starting to see a way forward here. I think I need to do the following:

  • Build a view to let registered users submit events. Allow the event description to be submitted in Markdown format, but warn the user that images won’t work.
  • In the admin interface, allow the admin to approve events by marking a boolean field in the database.
  • Build a custom admin view that contains a form that prompts the admin for their  Google password, and batch inserts all approved events to the Google calendar.

Initially I will support only non-repeating events. I’ll go back and try to add support for repeating events in “phase 2″ once I get the above functionality working.

I was looking through the Django docs and came across a new feature in trunk that allows you to extend the set of URLs supported by a model in the admin section. This will allow me to create a custom admin view to insert the events. I think this is just a new way of doing something that was already supported, and is described in the online Django book. I’ll have to update my working copy of Django trunk and try this out.

Tags: , , , ,

15 Feb 09 Event Calendar – Use Google Calendar?

I wrote a pretty complex event calendar for my PHP-Nuke site. You can find it here on sourceforge. It allows users to submit events for admin approval. Once approved they appear on the calendar. The application does all the calendar logic, including the drawing of the calendar. Repeating events were added, and they are pretty hairy as well. There are some pretty massively complicated SQL queries in that code, not to mention some crazy PHP logic to figure out if repeating events belong on a particular calendar view. It is also possible to make exceptions to repeating events.

I’ve been thinking about how to port this to my new Django powered site. One day I stumbled across the Google Calendar API, and I’m thinking about leveraging this. I will let Google handle all the crazy calendar logic, as they are no doubt better at it than I am. I will still let users submit events to the admin, but this time I will use the Google API to add events to a Google Calendar. Google can do the heavy lifting for me when it comes to displaying events and handling repeating events. Am I cheating by doing this? Ha. Well there are some advantages to this idea for my end users:

  • Anyone can display the site calendar on their blog or website. It won’t just be available on my site.
  • Users can use the RSS feed that Google generates. It was always on the to-do list for the existing calendar to add this feature, but I never got around to it.
  • If a user has a Google account of their own, they can add events from the new calendar to their own calendars.
  • Google can configured to notify users by email or SMS text messages of events.

So I’ve decided to explore this approach. If it doesn’t look like it will pan out, I will fall back to a more direct port of my old PHP code.

This approach isn’t without its own challenges. I can see right now I am going to have to think carefully about how to provide authentication to the Django code to allow it to add events to one of my Google calendars. I obviously don’t want to hard code my Google login and password in the Django application.

I think I will attack this problem in phases:

  1. Get a calendar to display on my site. This should be pretty easy; just use Google’s iframe code.
  2. Add an event through the Python API to gain experience with it. This is prototype, throw-away code.
  3. Build a front-end interface to let users create events, as was done in the PHP application.
  4. Think about an interface for letting users change or edit events. This could range from just sending an email to the admin, who could make the changes using the Google interface on one end of the spectrum, all the way to letting Python code programatically making the changes.
  5. Add support for repeating events.

I’ve already downloaded the Python version of the gdata library and have started reading the API docs. It looks interesting. Once I get familiar with this library, it looks like I could use it for other cool things like interfacing with YouTube and Google Docs.

Tags: , , , ,

25 Jan 09 Downloads: Uploading Files and Rating System

I’m very happy with the progress I’ve made on the downloads application. Again, this is similar to the Web links application, so it was somewhat familiar ground. Web links was one of the earliest applications I wrote, so this time I tried to do some slightly different things just to compare and contrast. For Web links, I used a base template for all of the Web links related sub-pages. This was primarily so I could include custom CSS in one spot, and build the main Web links navigation and search form in one place. For downloads, I instead used a template tag for this navigation area. I think I like this a little better, as now my views don’t have to construct and send in a search form to the template. This is tucked away in one spot in my template tag now. I still have to include the CSS on every page, but I don’t consider that a big deal.

Even though I have upload functionality in a few other places, I still got bit by a few gotchas when dealing with uploaded files. First of all, you have to construct your <form> tag with an additional encoding type attribute:

<form action="." method="post" enctype="multipart/form-data">

And on the server side, when validating the received POST arguments and files, you must bind the uploaded files to the form:

f = MyForm(request.POST, request.FILES)

Even though I’ve done this before multiple times, and it is clearly documented in the Django docs, I forgot these steps which caused a few minutes of confusion. I guess I am getting cocky now.

The really fun thing I got working is an AJAX style rating system to let people rate downloads. So now I have something similar to the YouTube.com “star” rating system. My progress was aided by this excellent tutorial at progressive-coding.com. The tutorial shows the steps to write the system in vanilla Javascript, then shows how to integrate it with several Javascript frameworks at the end. Since I’m using jQuery, I took the ideas there and wrote the whole thing with jQuery in mind, resulting in smaller code. I’m glad I took the time to study the tutorial and fashion it into my own making, as I learned a great deal by doing it. I was really impressed, again, by how easy jQuery makes these things. I will likely go over what I came up with in greater detail in a future post.

Now that I have this rating system in place, I think I will add a feature to limit the number of votes you can make on an item. I’m thinking of allowing a user to vote on a download once a week. It is so easy to vote now, just by a single click, that I’m sure people would be tempted to vote multiple times. I may go add something like this to the polls application too.

Tags: , ,

20 Jan 09 Weekend Coding Wrapup

I sat down and pounded out quite a bit of code last weekend, which is kind of unusual. I want to summarize what happened and leave some notes for myself.

On the comments front, I worked commenting into both the news and polls applications. This was astonishingly easy as I had already done the work once for the photo of the day. Django’s content types framework is really nice. Having the ability to attach comments to arbitrary models is just brilliant.

It occurred to me that I should take advantage of Django forms media, and then I wouldn’t have to repeat the several lines of <link> and <script> tags I have to add to every page that uses the markItUp! editor.  I added a “get_comment_form for” template tag, and then I thought I could just do a “{{ form.media }}” at the top of the template and then use the same form variable at the bottom of the template. But this did not work. I believe it has something to do with using the form variable in separate blocks in my templates that extend my base template. I posted a question about this on the django-user’s list, but did not receive a response. Oh well, not a huge deal, but I must investigate why this didn’t work sometime.

The other weird thing that happened was my Firefox browser started lurching to a halt when the shoutbox was being displayed with my new CSS scheme. I swear it didn’t do this before with the older CSS. So I got mad and disabled the Javascript scrolling for now, leaving it with a scroll bar (which actually doesn’t look that bad to me). I think this is only a problem on the Linux Firefox, I have not seen this on Windows. I have noticed that some sites give Linux Firefox some trouble if they have Javascript. It seems to eat an inordinate amount of CPU, making things like paging up and down a chore. I started looking at some jQuery-based tutorials for an alternate means for scrolling but didn’t find anything that jumped out at me right away. Again, I’ll put that on the to-do list.

On the CSS front, I came across the Blueprint CSS framework. This looks interesting for a design dolt like myself. It may help me with the overall site structure.

And finally last night I started working on the downloads application. This is my last “easy” application. After that I have to work on the calendar, the member map, Paypal, and the forums. The downloads application will be very similar to the Web links application that I did early on. It will contain many of the same types of views and sorting options. I am going to try and do it a bit smarter this time as I have learned a lot since then. I want to add an AJAX style star rating system and comments. I may then go back and refactor Web links with what I have learned. Ideally I should be looking for ways to share the code. If Django had a class-based way of handling views I think I could see a way forward. I could also finally break down and investigate generic views.

Tags: ,

16 Jan 09 CSS problems at bay for now…

I reworked the CSS on the test site by following this Jina Bolton tutorial at Sitepoint. My crazy div problem disappeared, which was my main goal. I consider this an interim step. I need something that looks like a semi-realistic website while I am developing. It helps me visualize what I am doing. I decided to leave the colors in the tutorial as-is rather than spend a lot of time tweaking them. Again, this is just a framework to get me going further. Hopefully I can enlist some of the folks on SG101 to assist with the final CSS and design.

The tutorial was very good. Since I’m not a designer, it was interesting to read how a good one works. And I learned a few things about CSS that should really help me in the future.

Speaking of cool CSS tricks, I found this tuturial on the web over lunch today. It is making me re-think how to markup the comments on my site. Semantically, perhaps I should display them as a list of blockquotes. Very interesting stuff!

Here is a screenshot of the test site with the borrowed CSS design.

Site with borrowed CSS

Tags: ,