msgbartop
by Brian Neal
msgbarbottom

15 Mar 09 Event Calendar: Everything But Repeating Events

Continuing along with the Google calendar integration, I built views to allow users to edit and delete the events they have previously submitted. These actions are simply requests; the admin still has to approve them. In the PHP version, these updates did not require admin approval. Here, they do, because the admin must supply the Google password to synchronize the events on the Google calendar. So the events simply get marked as “edit request” or “delete request”, which the admin must approve by toggling the status to “edit approved” or “delete approved”. The admin then visits the custom Google sync view, supplies the password, and the events are synchronized with Google via the gdata API. This seems to be working out well, although I’m not entirely happy with the workflow in the admin area to accomplish all this. Right now I have to visit the model change list, approve changes, then go to the custom Google sync view. We’ll see how this works out in practice.

I was very pleasantly surpised at how easy it was to add a link to my custom Google sync view in the admin area. I decided I wanted this link visible on all model views in the admin area (there is only one model right now: the event). I simply had to create this path in one of my template directories: admin/gcalendar/change_list.html (where gcalendar is the name of my application). Django will look to see if I have provided a change_list.html, and will use it in preference to the default one. And I didn’t even have to copy the whole template, I only needed to use template inheritance to override the block that provides the “object tools”. My change_list.html looks like this:

{% extends "admin/change_list.html" %}
{% block object-tools %}
{% if has_add_permission %}
<ul class="object-tools"><li><a href="add/{% if is_popup %}?_popup=1{% endif %}" class="addlink">Add {{ name }}</a></li>
<li><a href="google_sync/">Google Sync</a></li>
</ul>
{% endif %}
{% endblock %}

I just added the <li> and link to my “Google Sync” page. Here is a screen shot of my calendar event change list that shows this link in action. Check the upper right, next to the “Add” link. It was nice to see it was automatically styled to fit in with the rest of the admin.

GCalendar Admin Events Change List

Clicking on the link leads one to the custom admin view. Here, the list of approved events appears. The admin types in the Google password, and the view synchronizes (adds, updates, deletes) the events with Google. It might be a minor pain to enter your Google password every time, but I’d rather not hard-code or store the admin’s Google password in the database.

GCalendar Admin Google Sync View

I think using Google calendar is going to be cool. I like the fact that people can add the events to their own calendars, invite other people, and set email or SMS text message reminders for these events. I also like that other people can embed the site calendar on their own websites or blogs to spread the word.

With that in place, all I have left to do is to support repeating events. And actually, as it stands right now, I can also handle multi-day events. This covers 99% of the events on SurfGuitar101.com already. There are only two weekly gigs on the calendar right now. I have worked on this off and on for 3 or 4 weeks now, and have replicated most of the PHP version’s functionality by leveraging Google and Django. It took me about 3 or 4 months before I had this much working when I initially developed the PHP-Nuke version.

Repeating events might be difficult, because the gdata API sort of punts on them. In order to specify a repeating event, I have to format a fairly complex string according to some iCalendar RFC. Yikes. I will now poke around and see if there are some Python libraries available that I could use for this, or perhaps it isn’t as bad as I think it is and I can do it by hand. Or perhaps I will leave repeating events for the future since they aren’t used that much. I am itching to get what I have created so far deployed so I can start beta-testing and getting feedback from users.

Tags: , , , ,

08 Mar 09 Event Calendar: Submitting Events

I’m continuing on in my experiment to leverage Google calendar for my site’s event calendar. I have built a form that allows users to submit new events subject to admin approval. I modeled the form after the GUI used by Google calendar. I also decided to used jQuery UI’s datepicker. jQuery UI just released version 1.7, which is compatible with jQuery 1.3. After reading this blog entry, I’ve decided to try letting Google host jQuery for me. Google is also hosting the standard jQuery UI themes CSS files, so this seems awfully convenient.

Check out the screenshot of the event submit form:

Event Form with jQuery UI Datepicker

The datepicker is very useful, and will add a lot to the site, I think. My existing PHP application is using a much less elegant form. So far this form only handles non-repeating events. I will add support for those later.

I then turned my attention to the backend: how to synchronize my event models with Google calendar? It turns out there is already a Django application designed for this purpose: django-cal. It is designed in a very generic way and seems to be very well done. I decided I didn’t need its full functionality, but I did reference the source code when designing my back end. So a big thank-you to the creators of django-cal!

At this point I have enough functionality to add events and get them on the Google calendar. I have coded, but not tested, update and delete functionality. I am using the batch facility of the gdata API. In a typical work cycle, I need to approve anywhere from 1 to 7 or so events at once. This isn’t a lot of events, but I think doing it in a batch mode will speed a synchronization cycle up noticeably.

I store a “status” field in each event that keeps track of the event status: new, new and approved, edited, edited and approved, deleted, deleted and approved, and “on calendar”. When a user adds an event it becomes “new”. An admin reviews the event, and marks it “new and approved”. After synchronization, the status changes to “on calendar”. Likewise, my plan is to let users edit and request deletes for any events that they submit. My custom admin synchronization view will the perform the operations on all events with the “… and approved” status in a batch mode.

The custom admin view I wrote about in the last entry is also working well. I overrode the base django.contrib.admin template, so it looks well-integrated with the rest of the admin part of the site. I also figured out how to use the breadcrumbs for admin navigation, and noticed the base template will display a variable called “messages” with the nice green checkmark, if present. Likewise, the admin CSS has a class called “errorlist” for displaying errors. So it is quite easy to make a pretty professional looking custom admin screen by studying the base template and admin CSS files. I will post a screenshot of my custom admin view after I get some more events through the work cycle in a future blog post.

The Python gdata API isn’t as clumsy as I first thought it was. I was able to wrap it in a Calendar class to make it easier to use. If you need to update or delete an event in batch mode, it seems like you have to retrieve the event from Google first. I struggled against this initially, since I was storing away the “edit link” for each event I create. I was hoping to use the edit link to avoid another trip to Google for an update. But after searching the Google Calendar Data API group archives, it seems that is indeed recommended to retrieve any event prior to an update or delete, at least in batch mode.

This seems to be working out well so far. After I code and test the other operations I will blog with more details.

Tags: , , , ,

05 Mar 09 Custom Admin View Gotcha in Django 1.1

I ran into something that took a few hours to sort out, so I thought I would write it up here in case it would help some other poor dope out.

My goal is to create a custom admin view. I noticed that the Django docs had been updated recently, documenting a new member function called get_urls() on the ModelAdmin class. The docs are kind of sketchy right now, but I gathered that this was the new way to add custom admin views for a model. However, before you can use this new feature, you have to cut over to the new scheme of hooking the contrib.admin site into your project. This is what took me a few hours to figure out, as I hadn’t noticed this change in the docs.

This is how I had my URLConf in my Django 1.0 project:

urlpatterns = patterns('',
(r'^admin/(.*)', admin.site.root),
# ...

After updating my working copy of Django trunk, this is what my URLConf needed to look like to use the new admin get_urls() scheme:

urlpatterns = patterns('',
(r'^admin/', include(admin.site.urls)),
# ...

All the changes are on line 2. Notice, there are two changes you have to make:

  1. The admin.site.root function is deprecated. You should now use include(admin.site.urls).
  2. The regular expression has changed subtly. Notice that the capturing of the part of the URL after ‘admin/’ is no longer needed. I didn’t notice this right away, and trust me, it does weird things if you leave it in. :)

After finally figuring these two things out, I am happy to report that the get_urls() function on my ModelAdmin class worked as I expected. For example, if I use the regular expression ‘^my_view/$’ for my custom admin view, the view will be located at /admin/appname/modelname/my_view. This isn’t quite made crystal clear in the Django docs.

Now I am poised to add a custom admin view on my Google Calendar application. Stay tuned for more on that.

Tags: ,

21 Feb 09 Event Calendar – Experiments with Google Calendar

I haven’t had a lot of time to explore implementing an event calendar with Google Calendar, but I did try a few things out. First of all, it is pretty easy to embed a Google Calendar into your website:

Google Calendar Embedded on Website

Google Calendar Embedded on Website

As I mentioned in the last post, I downloaded the Google Python gdata client library. I wrote a small Python script based on the provided example code. I was able to add events to my calendar from this Python script:

import gdata.calendar
import gdata.calendar.service
import gdata.service
import atom
import atom.service
import time

cal_client = gdata.calendar.service.CalendarService()
cal_client.email = 'xxx@gmail.com'
cal_client.password = 'xxx'
cal_client.source = 'Google-Calendar_Python_Sample-1.0'
cal_client.ProgrammaticLogin()

title = 'My Test Event from Python'
content = '''This is the content of the event in some fashion.
<a href="http://google.com">My Link</a>

http://surfguitar101.com.

<b>Bold</b> text.
<p>Here is a paragraph.</p>
<p>And another</p>
<ul><li>Item 1</li><li>Item 2</li></ul>
<img src="http://surfguitar101.com/modules/Forums/images/smiles/icon_cool.gif" alt="Smiley" />
'''
where = 'An undisclosed location.'
#cal_path = '/calendar/feeds/default/private/full'
cal_path = '/calendar/feeds/xxx@group.calendar.google.com/private/full'

event = gdata.calendar.CalendarEventEntry()
event.title = atom.Title(text=title)
event.content = atom.Content(text=content)
event.where.append(gdata.calendar.Where(value_string=where))

# Use current time for the start_time and have the event last 1 hour
start_time = time.strftime('%Y-%m-%dT%H:%M:%S.000Z', time.gmtime())
end_time = time.strftime('%Y-%m-%dT%H:%M:%S.000Z', time.gmtime(time.time() + 3600))
event.when.append(gdata.calendar.When(start_time=start_time, end_time=end_time))

new_event = cal_client.InsertEvent(event, cal_path)

As you can see, I was testing to see if I could insert HTML into the event, and if it would display correctly on the Google Calendar. According to my tests, it does appear you can insert simple HTML. The only thing that did not work was <img> tags. Unfortunately, if you log into your Google account, and then go to your Google Calendar and try to edit this event using the Google GUI, all HTML will be stripped. This will make it difficult to edit an existing event and will probably steer me towards keeping the user added event around in the database for future editing until that event has expired.

I have to say I am not really impressed with the Python API. It seems rather “un-Pythonic”. Why do I have to construct atom objects and assign them to the event title and content fields? The API should do this for me. Why do I have to format my own date strings? The API should accept Python datetime objects and do that for me. Pretty disappointing coming from Google.

I am starting to see a way forward here. I think I need to do the following:

  • Build a view to let registered users submit events. Allow the event description to be submitted in Markdown format, but warn the user that images won’t work.
  • In the admin interface, allow the admin to approve events by marking a boolean field in the database.
  • Build a custom admin view that contains a form that prompts the admin for their  Google password, and batch inserts all approved events to the Google calendar.

Initially I will support only non-repeating events. I’ll go back and try to add support for repeating events in “phase 2″ once I get the above functionality working.

I was looking through the Django docs and came across a new feature in trunk that allows you to extend the set of URLs supported by a model in the admin section. This will allow me to create a custom admin view to insert the events. I think this is just a new way of doing something that was already supported, and is described in the online Django book. I’ll have to update my working copy of Django trunk and try this out.

Tags: , , , ,

15 Feb 09 Event Calendar – Use Google Calendar?

I wrote a pretty complex event calendar for my PHP-Nuke site. You can find it here on sourceforge. It allows users to submit events for admin approval. Once approved they appear on the calendar. The application does all the calendar logic, including the drawing of the calendar. Repeating events were added, and they are pretty hairy as well. There are some pretty massively complicated SQL queries in that code, not to mention some crazy PHP logic to figure out if repeating events belong on a particular calendar view. It is also possible to make exceptions to repeating events.

I’ve been thinking about how to port this to my new Django powered site. One day I stumbled across the Google Calendar API, and I’m thinking about leveraging this. I will let Google handle all the crazy calendar logic, as they are no doubt better at it than I am. I will still let users submit events to the admin, but this time I will use the Google API to add events to a Google Calendar. Google can do the heavy lifting for me when it comes to displaying events and handling repeating events. Am I cheating by doing this? Ha. Well there are some advantages to this idea for my end users:

  • Anyone can display the site calendar on their blog or website. It won’t just be available on my site.
  • Users can use the RSS feed that Google generates. It was always on the to-do list for the existing calendar to add this feature, but I never got around to it.
  • If a user has a Google account of their own, they can add events from the new calendar to their own calendars.
  • Google can configured to notify users by email or SMS text messages of events.

So I’ve decided to explore this approach. If it doesn’t look like it will pan out, I will fall back to a more direct port of my old PHP code.

This approach isn’t without its own challenges. I can see right now I am going to have to think carefully about how to provide authentication to the Django code to allow it to add events to one of my Google calendars. I obviously don’t want to hard code my Google login and password in the Django application.

I think I will attack this problem in phases:

  1. Get a calendar to display on my site. This should be pretty easy; just use Google’s iframe code.
  2. Add an event through the Python API to gain experience with it. This is prototype, throw-away code.
  3. Build a front-end interface to let users create events, as was done in the PHP application.
  4. Think about an interface for letting users change or edit events. This could range from just sending an email to the admin, who could make the changes using the Google interface on one end of the spectrum, all the way to letting Python code programatically making the changes.
  5. Add support for repeating events.

I’ve already downloaded the Python version of the gdata library and have started reading the API docs. It looks interesting. Once I get familiar with this library, it looks like I could use it for other cool things like interfacing with YouTube and Google Docs.

Tags: , , , ,

14 Feb 09 Member Map Part 2

I’ve gotten the Member Map application ported to my Django powered site now. This was pretty straightforward and a lot of fun because of the Javascript aspect. Let me address the points I made in part 1 of this post, below.

  1. Leveraging jQuery. I did make use of jQuery for the client side Javascript. This really did simplify things. In the old version of the code, I had a 282 line Javascript file. The new version weighs in at 161 lines, which is 57% the size of the earlier version.
  2. Use of JSON encoder library. It turns out that Django includes a version of the simplejson library already. This is used in Django’s serialization feature. I probably could have used this feature, but I didn’t need to serialize an entire model, and wanted to rename some of the fields. So I simply did a “import django.utils.simplejson as json” and went to town. My server side code was now much simpler than the previous version where I encoded by hand.
  3. I did continue to use the markItUp! editor to let users edit their message to go along with their location. I learned that the size of the editor is easily adjusted through the use of CSS. So I was able to override the default size of the editor by including my own CSS style sheet below the default sheet.
  4. Reducing complexity. Here is where it got interesting. I decided to add a “json” field to the MapEntry model. This field is populated whenever a model object is saved by providing my own save() method. Inside my save() I use Django’s template render_to_string() to render the HTML  message that appears in the pop-up balloon on the map. This consists  of the user’s avatar followed by a message that is marked up in Markdown. I then build a JSON representation of the MapEntry using Django’s simplejson. This representation consists of the following fields: name, location, latitude, longitude, and the HTML message. By pre-saving all of this information, it makes it much more efficient to retrieve 200+ users’ information when the map is first loaded. In the view function, I make use of the values_list() queryset method to retrieve all the json fields. I don’t need the ORM to convert each MapEntry to an actual Python object.

Another thing I did differently for the port was when a user updates or adds their position on the map, I don’t reload the entire map. I just adjust their information to reduce bandwidth with an AJAX post. I’m not sure why I didn’t do this before, perhaps because it was to tedious to do without jQuery. It was just easier to reload the whole thing.

This was all well and cool, doing all this pre-computation, but what happens when a user changes her avatar? Her entry on the map will likely have a broken graphic. No problem. Here I took advantage of Django’s signals. I attached a signal handler to listen for changes to the UserProfile model. Whenever the UserProfile is saved, my signal handler runs and re-saves the corresponding MapEntry to regenerate the JSON. Very slick.

And finally, another thing that I learned about Django was that simple_tags can have defaulted arguments. I added an argument to my avatar tag so that I could apply CSS to the generated HTML img tag if needed. In the Member Map pop-up balloon, it looked nicer if the avatar was floated to the left.

So all and all, again, I’m very happy with how easy Django makes writing an application like this. Using jQuery was also a big productivity booster.

Here are some screenshots. You can also see the new Blueprints CSS in action as well.

Member Map

Member Map 1

Member Map

Member Map 2

Tags: , , , ,

06 Feb 09 Member Map Part 1

I started working on the Member Map application last weekend. This will be a port of the PHP-Nuke module I wrote, which you can find here. In a nutshell, this application displays a Google Map on your site, and allows site members to place markers representing their location on the map. If you click on a marker, a balloon pops up which displays the user’s avatar, a link to their profile, and a short message that they have typed previously. The Nuke version of this application is a big hit on the current site, and last time I checked, we had over 220 members on the map.

The existing application consists of the PHP-Nuke code, written in PHP, on the server side, and a fair amount of Javascript on the client side. All interaction with Google Maps is done through the Javascript code.

For the port to Django, I want to do a couple of things different.

  1. Leverage jQuery in the client-side Javascript. The existing version was written before I was aware of Javascript libraries, and it is all “vanilla” Javascript. By using jQuery, the amount of code I have to write will be much smaller. And by taking advantage of jQuery, it should be more portable and browser robust.
  2. On the server side, the existing application did not use any JSON encoder library. I simply constructed the JSON by hand. I can’t remember why I did this. I think it was partly because I didn’t know any better, and partly because the PHP version I had on the original hosting server may not have had it. I will explore what Python and/or Django options exist.
  3. I’ll leverage the markItUp! editor that I’ve been using for comments to let the users enter Markdown comments. I’ve used this in the last couple of apps I’ve written and it will be easy to implement. The only concern with doing this is the user may be tempted to write a lot more than in the older app, and it may make for large “balloons”. We’ll see.
  4. I want to make the new application less database and computationally intense than the older version. I don’t have any problems running the current version with 200 users, but I think I could do it a bit smarter this time around. This means I am going to try to save marked up and JSON encoded strings in the database rather than compute them on every page load. I am also wary of instantiating 200+ map entry model objects with the Django ORM (albeit in a loop) just to generate the JSON. Luckily Django lets you use raw SQL when needed. But in this case I think a simple use of the values() method will really be useful here.

So, those are my going in goals. I’ve already started working on this, and I’ve learned a lot so far. In the next post or two I will detail the progress and what problems I ran into.

Tags: , , , ,

25 Jan 09 Displaying a Page’s SQL Queries

I’ve been meaning to start paying attention to the SQL queries that Django is generating for me. I’d like to know if there are any dumb things I am doing. And down the road I am going to start employing low-level caching, and seeing what queries I am doing will help me decide what things to cache. I finally broke down and figured out how you can display the SQL queries used during an HTTP request on the rendered web page. Django makes this quite easy and it is fully documented.

If your TEMPLATE_CONTEXT_PROCESSORS in your settings.py file contains “django.core.context_processors.debug”, and DEBUG is set to True, you have access to a variable in your templates called sql_queries. This is an ordered list of {'sql': ..., 'time': ...} dictionaries that represent every SQL query and how long each one took during the processing of the request.

It was quite easy to add the following to the very bottom of my site’s base template:

{% if debug %}
<div id="debug">
<ol>
{% for s in sql_queries %}
<li>{{ s.sql }} : <b>({{ s.time }})</b></li>
{% endfor %}
</ol>
</div>
{% endif %}

Now I can keep an eye on what’s happening at the database level. I’ve already added a few key “select_related” clauses to some of my Django ORM queries that have saved 10-30% on the number of trips to the database per page. Cool!

Tags: , ,

25 Jan 09 Downloads: Uploading Files and Rating System

I’m very happy with the progress I’ve made on the downloads application. Again, this is similar to the Web links application, so it was somewhat familiar ground. Web links was one of the earliest applications I wrote, so this time I tried to do some slightly different things just to compare and contrast. For Web links, I used a base template for all of the Web links related sub-pages. This was primarily so I could include custom CSS in one spot, and build the main Web links navigation and search form in one place. For downloads, I instead used a template tag for this navigation area. I think I like this a little better, as now my views don’t have to construct and send in a search form to the template. This is tucked away in one spot in my template tag now. I still have to include the CSS on every page, but I don’t consider that a big deal.

Even though I have upload functionality in a few other places, I still got bit by a few gotchas when dealing with uploaded files. First of all, you have to construct your <form> tag with an additional encoding type attribute:

<form action="." method="post" enctype="multipart/form-data">

And on the server side, when validating the received POST arguments and files, you must bind the uploaded files to the form:

f = MyForm(request.POST, request.FILES)

Even though I’ve done this before multiple times, and it is clearly documented in the Django docs, I forgot these steps which caused a few minutes of confusion. I guess I am getting cocky now.

The really fun thing I got working is an AJAX style rating system to let people rate downloads. So now I have something similar to the YouTube.com “star” rating system. My progress was aided by this excellent tutorial at progressive-coding.com. The tutorial shows the steps to write the system in vanilla Javascript, then shows how to integrate it with several Javascript frameworks at the end. Since I’m using jQuery, I took the ideas there and wrote the whole thing with jQuery in mind, resulting in smaller code. I’m glad I took the time to study the tutorial and fashion it into my own making, as I learned a great deal by doing it. I was really impressed, again, by how easy jQuery makes these things. I will likely go over what I came up with in greater detail in a future post.

Now that I have this rating system in place, I think I will add a feature to limit the number of votes you can make on an item. I’m thinking of allowing a user to vote on a download once a week. It is so easy to vote now, just by a single click, that I’m sure people would be tempted to vote multiple times. I may go add something like this to the polls application too.

Tags: , ,

20 Jan 09 Weekend Coding Wrapup

I sat down and pounded out quite a bit of code last weekend, which is kind of unusual. I want to summarize what happened and leave some notes for myself.

On the comments front, I worked commenting into both the news and polls applications. This was astonishingly easy as I had already done the work once for the photo of the day. Django’s content types framework is really nice. Having the ability to attach comments to arbitrary models is just brilliant.

It occurred to me that I should take advantage of Django forms media, and then I wouldn’t have to repeat the several lines of <link> and <script> tags I have to add to every page that uses the markItUp! editor.  I added a “get_comment_form for” template tag, and then I thought I could just do a “{{ form.media }}” at the top of the template and then use the same form variable at the bottom of the template. But this did not work. I believe it has something to do with using the form variable in separate blocks in my templates that extend my base template. I posted a question about this on the django-user’s list, but did not receive a response. Oh well, not a huge deal, but I must investigate why this didn’t work sometime.

The other weird thing that happened was my Firefox browser started lurching to a halt when the shoutbox was being displayed with my new CSS scheme. I swear it didn’t do this before with the older CSS. So I got mad and disabled the Javascript scrolling for now, leaving it with a scroll bar (which actually doesn’t look that bad to me). I think this is only a problem on the Linux Firefox, I have not seen this on Windows. I have noticed that some sites give Linux Firefox some trouble if they have Javascript. It seems to eat an inordinate amount of CPU, making things like paging up and down a chore. I started looking at some jQuery-based tutorials for an alternate means for scrolling but didn’t find anything that jumped out at me right away. Again, I’ll put that on the to-do list.

On the CSS front, I came across the Blueprint CSS framework. This looks interesting for a design dolt like myself. It may help me with the overall site structure.

And finally last night I started working on the downloads application. This is my last “easy” application. After that I have to work on the calendar, the member map, Paypal, and the forums. The downloads application will be very similar to the Web links application that I did early on. It will contain many of the same types of views and sorting options. I am going to try and do it a bit smarter this time as I have learned a lot since then. I want to add an AJAX style star rating system and comments. I may then go back and refactor Web links with what I have learned. Ideally I should be looking for ways to share the code. If Django had a class-based way of handling views I think I could see a way forward. I could also finally break down and investigate generic views.

Tags: ,