<?xml version="1.0" encoding="UTF-8"?>
<feed
  xmlns="http://www.w3.org/2005/Atom"
  xmlns:thr="http://purl.org/syndication/thread/1.0"
  xml:lang="en"
   >
  <title type="text">Death of a Gremmie</title>
  <subtitle type="text">Brian Neal's blog about programming.</subtitle>

  <updated>2012-05-13T18:30:33Z</updated>
  <generator uri="http://blogofile.com/">Blogofile</generator>

  <link rel="alternate" type="text/html" href="http://deathofagremmie.com" />
  <id>http://deathofagremmie.com/feed/atom/</id>
  <link rel="self" type="application/atom+xml" href="http://deathofagremmie.com/feed/atom/" />
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Fixed pages for Django]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2012/05/13/fixed-pages-for-django" />
    <id>http://deathofagremmie.com/2012/05/13/fixed-pages-for-django</id>
    <updated>2012-05-13T13:30:00Z</updated>
    <published>2012-05-13T13:30:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Django" />
    <summary type="html"><![CDATA[Fixed pages for Django]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2012/05/13/fixed-pages-for-django"><![CDATA[<div class="document">
<p>I have been using Django's <a class="reference external" href="https://docs.djangoproject.com/en/1.4/ref/contrib/flatpages/">flatpages app</a> for some simple, static pages that
were supposed to be temporary. I slapped a Javascript editor on the admin page
and it has worked very well. However some of the pages have long outlived their
&quot;temporary&quot; status, and I find myself needing to update them. It is then that I
get angry at the Javascript editor, and there is no way to keep any kind of
history on the page without having to fish through old database backups. I
started to think it would be nice to write the content in a nice markup
language, for example <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a>, which I could then commit to version
control. I would just need a way to generate HTML from the source text to
produce the flatpage content.</p>
<p>Of course I could use the template filters in <a class="reference external" href="https://docs.djangoproject.com/en/1.4/ref/contrib/markup/">django.contrib.markup</a>. But
turning markup into HTML at page request time can be more expensive than I like.
Yes, I could cache the page, but I'd like the process to be more explicit.</p>
<p>In my first attempt at doing this, I wrote a <a class="reference external" href="https://docs.djangoproject.com/en/1.4/howto/custom-management-commands/">custom management command</a> that
used a dictionary in my <tt class="docutils literal">settings.py</tt> file to map reStructuredText files to
flatpage URLs. My management command would open the input file, convert it to
HTML, then find the <tt class="docutils literal">FlatPage</tt> object associated with the URL. It would then
update the object with the new HTML content and save it.</p>
<p>This worked okay, but in the end I decided that the pages I wanted to update
were not temporary, quick &amp; dirty pages, which is kind of how I view flatpages.
So I decided to stop leaning on the flatpages app for these pages.</p>
<p>I then modified the management command to read a given input file, convert it
to an HTML fragment, then save it in my templates directory. Thus, a file stored
in my project directory as <tt class="docutils literal">fixed/about.rst</tt> would get transformed to
<tt class="docutils literal">templates/fixed/about.html</tt>. Here is the source to the command which I saved
as <tt class="docutils literal">make_fixed_page.py</tt>:</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">os.path</span>
<span class="kn">import</span> <span class="nn">glob</span>

<span class="kn">import</span> <span class="nn">docutils.core</span>
<span class="kn">from</span> <span class="nn">django.core.management.base</span> <span class="kn">import</span> <span class="n">LabelCommand</span><span class="p">,</span> <span class="n">CommandError</span>
<span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>


<span class="k">class</span> <span class="nc">Command</span><span class="p">(</span><span class="n">LabelCommand</span><span class="p">):</span>
    <span class="n">help</span> <span class="o">=</span> <span class="s">&quot;Generate HTML from restructured text files&quot;</span>
    <span class="n">args</span> <span class="o">=</span> <span class="s">&quot;&lt;inputfile1&gt; &lt;inputfile2&gt; ... | all&quot;</span>

    <span class="k">def</span> <span class="nf">handle_label</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Process input file(s)&quot;&quot;&quot;</span>

        <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="s">&#39;PROJECT_PATH&#39;</span><span class="p">):</span>
            <span class="k">raise</span> <span class="n">CommandError</span><span class="p">(</span><span class="s">&quot;Please add a PROJECT_PATH setting&quot;</span><span class="p">)</span>

        <span class="bp">self</span><span class="o">.</span><span class="n">src_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">settings</span><span class="o">.</span><span class="n">PROJECT_PATH</span><span class="p">,</span> <span class="s">&#39;fixed&#39;</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">dst_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">settings</span><span class="o">.</span><span class="n">PROJECT_PATH</span><span class="p">,</span> <span class="s">&#39;templates&#39;</span><span class="p">,</span> <span class="s">&#39;fixed&#39;</span><span class="p">)</span>

        <span class="k">if</span> <span class="n">filename</span> <span class="o">==</span> <span class="s">&#39;all&#39;</span><span class="p">:</span>
            <span class="n">files</span> <span class="o">=</span> <span class="n">glob</span><span class="o">.</span><span class="n">glob</span><span class="p">(</span><span class="s">&quot;</span><span class="si">%s%s</span><span class="s">*.rst&quot;</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">src_dir</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">sep</span><span class="p">))</span>
            <span class="n">files</span> <span class="o">=</span> <span class="p">[</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">files</span><span class="p">]</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="n">files</span> <span class="o">=</span> <span class="p">[</span><span class="n">filename</span><span class="p">]</span>

        <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">process_page</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">process_page</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Processes one fixed page&quot;&quot;&quot;</span>

        <span class="c"># retrieve source text</span>
        <span class="n">src_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">src_dir</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">src_path</span><span class="p">,</span> <span class="s">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
                <span class="n">src_text</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
        <span class="k">except</span> <span class="ne">IOError</span><span class="p">,</span> <span class="n">ex</span><span class="p">:</span>
            <span class="k">raise</span> <span class="n">CommandError</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">ex</span><span class="p">))</span>

        <span class="c"># transform text</span>
        <span class="n">content</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">transform_input</span><span class="p">(</span><span class="n">src_text</span><span class="p">)</span>

        <span class="c"># write output</span>
        <span class="n">basename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">filename</span><span class="p">))[</span><span class="mi">0</span><span class="p">]</span>
        <span class="n">dst_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dst_dir</span><span class="p">,</span> <span class="s">&#39;</span><span class="si">%s</span><span class="s">.html&#39;</span> <span class="o">%</span> <span class="n">basename</span><span class="p">)</span>

        <span class="k">try</span><span class="p">:</span>
            <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">dst_path</span><span class="p">,</span> <span class="s">&#39;w&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
                <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">content</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">))</span>
        <span class="k">except</span> <span class="ne">IOError</span><span class="p">,</span> <span class="n">ex</span><span class="p">:</span>
            <span class="k">raise</span> <span class="n">CommandError</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">ex</span><span class="p">))</span>

        <span class="n">prefix</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">commonprefix</span><span class="p">([</span><span class="n">src_path</span><span class="p">,</span> <span class="n">dst_path</span><span class="p">])</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">&quot;</span><span class="si">%s</span><span class="s"> -&gt; </span><span class="si">%s</span><span class="se">\n</span><span class="s">&quot;</span> <span class="o">%</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">dst_path</span><span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">prefix</span><span class="p">):]))</span>

    <span class="k">def</span> <span class="nf">transform_input</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">src_text</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;Transforms input restructured text to HTML&quot;&quot;&quot;</span>

        <span class="k">return</span> <span class="n">docutils</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">publish_parts</span><span class="p">(</span><span class="n">src_text</span><span class="p">,</span> <span class="n">writer_name</span><span class="o">=</span><span class="s">&#39;html&#39;</span><span class="p">,</span>
                <span class="n">settings_overrides</span><span class="o">=</span><span class="p">{</span>
                    <span class="s">&#39;doctitle_xform&#39;</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span>
                    <span class="s">&#39;initial_header_level&#39;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
                    <span class="p">})[</span><span class="s">&#39;html_body&#39;</span><span class="p">]</span>
</pre></div>
<p>Next I would need a template that could render these fragments. I remembered
that the Django <a class="reference external" href="https://docs.djangoproject.com/en/1.4/ref/templates/builtins/#include">include tag</a> could take a variable as an argument. Thus I
could create a single template that could render all of these &quot;fixed&quot; pages.
Here is the template <tt class="docutils literal">templates/fixed/base.html</tt>:</p>
<pre class="literal-block">
{% extends 'base.html' %}
{% block title %}{{ title }}{% endblock %}
{% block content %}
{% include content_template %}
{% endblock %}
</pre>
<p>I just need to pass in <tt class="docutils literal">title</tt> and <tt class="docutils literal">content_template</tt> context variables. The
latter will control which HTML fragment I include.</p>
<p>I then turned to the view function which would render this template. I wanted to
make this as generic and easy to do as possible. Since I was abandoning
flatpages, I would need to wire these up in my <tt class="docutils literal">urls.py</tt>. At first I didn't
think I could use Django's new <a class="reference external" href="https://docs.djangoproject.com/en/1.4/topics/class-based-views/">class-based generic views</a> for this, but after
some fiddling around, I came up with a very nice solution:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">django.views.generic</span> <span class="kn">import</span> <span class="n">TemplateView</span>

<span class="k">class</span> <span class="nc">FixedView</span><span class="p">(</span><span class="n">TemplateView</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    For displaying our &quot;fixed&quot; views generated with the custom command</span>
<span class="sd">    make_fixed_page.</span>

<span class="sd">    &quot;&quot;&quot;</span>
    <span class="n">template_name</span> <span class="o">=</span> <span class="s">&#39;fixed/base.html&#39;</span>
    <span class="n">title</span> <span class="o">=</span> <span class="s">&#39;&#39;</span>
    <span class="n">content_template</span> <span class="o">=</span> <span class="s">&#39;&#39;</span>

    <span class="k">def</span> <span class="nf">get_context_data</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
        <span class="n">context</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">FixedView</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">get_context_data</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
        <span class="n">context</span><span class="p">[</span><span class="s">&#39;title&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">title</span>
        <span class="n">context</span><span class="p">[</span><span class="s">&#39;content_template&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">content_template</span>
        <span class="k">return</span> <span class="n">context</span>
</pre></div>
<p>This allowed me to do the following in my <tt class="docutils literal">urls.py</tt> file:</p>
<div class="highlight"><pre><span class="n">urlpatterns</span> <span class="o">=</span> <span class="n">patterns</span><span class="p">(</span><span class="s">&#39;&#39;</span><span class="p">,</span>
   <span class="c"># ...</span>

   <span class="n">url</span><span class="p">(</span><span class="s">r&#39;^about/$&#39;</span><span class="p">,</span>
       <span class="n">FixedView</span><span class="o">.</span><span class="n">as_view</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s">&#39;About&#39;</span><span class="p">,</span> <span class="n">content_template</span><span class="o">=</span><span class="s">&#39;fixed/about.html&#39;</span><span class="p">),</span>
       <span class="n">name</span><span class="o">=</span><span class="s">&#39;about&#39;</span><span class="p">),</span>
   <span class="n">url</span><span class="p">(</span><span class="s">r&#39;^colophon/$&#39;</span><span class="p">,</span>
       <span class="n">FixedView</span><span class="o">.</span><span class="n">as_view</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="s">&#39;Colophon&#39;</span><span class="p">,</span> <span class="n">content_template</span><span class="o">=</span><span class="s">&#39;fixed/colophon.html&#39;</span><span class="p">),</span>
       <span class="n">name</span><span class="o">=</span><span class="s">&#39;colophon&#39;</span><span class="p">),</span>

   <span class="c"># ...</span>
</pre></div>
<p>Now I have a way to efficiently serve reStructuredText files as &quot;fixed pages&quot;
that I can put under source code control.</p>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Mounting a Synology DiskStation on Ubuntu 12.04]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2012/05/01/mounting-a-synology-diskstation-on-ubuntu-12.04" />
    <id>http://deathofagremmie.com/2012/05/01/mounting-a-synology-diskstation-on-ubuntu-12.04</id>
    <updated>2012-05-01T20:45:00Z</updated>
    <published>2012-05-01T20:45:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Ubuntu" />
    <category scheme="http://deathofagremmie.com" term="Synology" />
    <category scheme="http://deathofagremmie.com" term="Linux" />
    <summary type="html"><![CDATA[Mounting a Synology DiskStation on Ubuntu 12.04]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2012/05/01/mounting-a-synology-diskstation-on-ubuntu-12.04"><![CDATA[<div class="document">
<p>I have a Synology DiskStation that I use for a home NAS. I didn't take good
notes when I got it to work with Ubuntu 10.04, so I had to fumble about when I
upgraded to Ubuntu 12.04 last weekend. So for next time, here is the recipe I
used.</p>
<p>First, you need to install the <strong>cifs-utils</strong> package:</p>
<pre class="literal-block">
$ sudo apt-get install cifs-utils
</pre>
<p>Next, I added the following text to my <tt class="docutils literal">/etc/fstab</tt> file:</p>
<pre class="literal-block">
//192.168.1.2/shared /mnt/syn cifs noauto,iocharset=utf8,uid=1000,gid=1000,credentials=/home/brian/.cifspwd 0 0
</pre>
<p>Take note of the following:</p>
<ul class="simple">
<li>Replace <tt class="docutils literal">//192.168.1.2/</tt> with the IP address or hostname of your
DiskStation. Likewise, <tt class="docutils literal">/shared</tt> is just the path on the DiskStation that I
wanted to mount.</li>
<li><tt class="docutils literal">/mnt/syn</tt> is the mount point where the DiskStation will appear on our local
filesystem.</li>
<li>I didn't want my laptop to mount the DiskStation on bootup, so I used the
<tt class="docutils literal">noauto</tt> parameter.</li>
<li>The <tt class="docutils literal">uid</tt> and <tt class="docutils literal">gid</tt> should match the user and group IDs of your Ubuntu
user. You can find this by grepping your username in <tt class="docutils literal">/etc/passwd</tt>.</li>
<li>The <tt class="docutils literal">credentials</tt> parameter should point to a file you create that contains
the username and password of the DiskStation user you want to impersonate (see
below).</li>
</ul>
<p>Your <tt class="docutils literal">.cifspwd</tt> file should look like the following:</p>
<pre class="literal-block">
username=username
password=password
</pre>
<p>Obviously you'll want to use the real username / password pair of a user on your
DiskStation.</p>
<p>To be paranoid, you should make the file owned by root and readable only by
root. Do this after you get everything working:</p>
<pre class="literal-block">
$ sudo chown root:root .cifspwd
$ sudo chmod 0600 .cifspwd
</pre>
<p>I created the mount point for the DiskStation with:</p>
<pre class="literal-block">
$ sudo mkdir -p /mnt/syn
</pre>
<p>Then, whenever I want to use the DiskStation I use:</p>
<pre class="literal-block">
$ sudo mount /mnt/syn
</pre>
<p>And to unmount it:</p>
<pre class="literal-block">
$ sudo umount /mnt/syn
</pre>
<p>You can avoid the <tt class="docutils literal">mount</tt> and <tt class="docutils literal">umount</tt> commands if you remove the
<tt class="docutils literal">noauto</tt> parameter from the <tt class="docutils literal">/etc/fstab</tt> entry. In that case, Ubuntu will
automatically try to mount the DiskStation at startup.</p>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Upgrading to Django 1.4]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2012/04/15/upgrading-to-django-1.4" />
    <id>http://deathofagremmie.com/2012/04/15/upgrading-to-django-1.4</id>
    <updated>2012-04-15T14:50:00Z</updated>
    <published>2012-04-15T14:50:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Django" />
    <summary type="html"><![CDATA[Upgrading to Django 1.4]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2012/04/15/upgrading-to-django-1.4"><![CDATA[<div class="document">
<p><a class="reference external" href="https://www.djangoproject.com/weblog/2012/mar/23/14/">Django 1.4</a> came out recently, and I took a few hours to upgrade my first site
yesterday. I thought it would be useful for my own reference to write down what
I did. I hope it will be useful to others. I'd love to read what you had to do,
so if you went through this process and blogged about it, please leave a
comment. Please keep in mind these aren't hard and fast steps or a recipe to
follow, as my sites are probably nothing like yours and may use different
features of Django.</p>
<div class="section" id="preparation">
<h3>Preparation</h3>
<p>The first thing I did was to read very carefully the <a class="reference external" href="https://docs.djangoproject.com/en/1.4/releases/1.4/">Django 1.4 release
notes</a>. The Django team does a great job of documenting what has changed, so it
is well worth your time to read the release notes. It is also a good idea to at
least skim the <a class="reference external" href="https://docs.djangoproject.com/en/1.4/internals/deprecation/">Django Deprecation Timeline</a>. After reading these, you should
make a list of the things you want to change, add, or remove.</p>
</div>
<div class="section" id="tips">
<h3>Tips</h3>
<p>After deciding what areas you want or need to change in your code, these tips
may be useful to help you implement the changes.</p>
<ol class="arabic simple">
<li><strong>Run with warnings turned on</strong>. Use this command to run the development
server: <tt class="docutils literal">$ python <span class="pre">-Wall</span> manage.py runserver</tt>. Django makes use of <a class="reference external" href="http://docs.python.org/library/warnings.html">Python's
warning system</a> to flag features that are deprecated. By running Python with
the <tt class="docutils literal"><span class="pre">-Wall</span></tt> switch, you'll see these warnings in the development server
output.</li>
<li><strong>Use the debugger to track down warnings</strong>. Not sure where a pesky warning
is coming from? Just open the Django source code in your editor and put a
<tt class="docutils literal">import pdb; pdb.set_trace()</tt> line right above or below the warning. You
can then use the debugger's <tt class="docutils literal">w</tt> command to get a stack trace and find out
exactly what code is leading to the warning. In my case I kept getting a few
warnings with no idea where they were coming from. I used this technique to
verify the warnings were coming from third party code and not my own. For
more information on using the debugger (and you really <strong>should</strong> know how to
use this invaluable tool), see the <a class="reference external" href="http://docs.python.org/library/pdb.html">Pdb documentation</a>.</li>
</ol>
</div>
<div class="section" id="my-upgrade-experience">
<h3>My upgrade experience</h3>
<p>Here is a list of things that I did during my port. Again, you may not need to
do these, and the next site I upgrade may have a different list. All of these
changes (except for the first) are described in the <a class="reference external" href="https://docs.djangoproject.com/en/1.4/releases/1.4/">Django 1.4 release notes</a>.</p>
<ol class="arabic simple">
<li><strong>Upgrade my Django debug toolbar</strong>. As of this writing, the Django debug
toolbar I got from PyPI was not compatible with Django 1.4. I simply
uninstalled it and grabbed the development version from GitHub with
<tt class="docutils literal">pip install <span class="pre">git+https://github.com/django-debug-toolbar/django-debug-toolbar.git</span></tt>.</li>
<li><strong>Remove the ADMIN_MEDIA_PREFIX setting</strong>. The admin application in
Django 1.4 now relies on the <tt class="docutils literal">staticfiles</tt> application (introduced in
Django 1.3) to handle the serving of static assets.</li>
<li><strong>Remove use of the {% admin_media_prefix %} template tag</strong>. Related to the
above, this tag is now deprecated. I had a custom admin view that used this
template tag, and I simply replaced it with <tt class="docutils literal">{{ STATIC_URL <span class="pre">}}/admin</span></tt>.</li>
<li><strong>Remove verify_exists on URLFields</strong>. The <tt class="docutils literal">verify_exists</tt> option to
the <tt class="docutils literal">URLField</tt> has been removed for performance and security reasons. I had
always set this to <tt class="docutils literal">False</tt>; now I just had to remove it altogether.</li>
<li><strong>Add the require_debug_false filter to logging settings</strong>. As explained in
the release notes, this change prevents admin error emails from being sent
while in <tt class="docutils literal">DEBUG</tt> mode.</li>
<li><strong>django.conf.urls.defaults is deprecated</strong>. I changed my imports in all
<tt class="docutils literal">urls.py</tt> files to use <tt class="docutils literal">django.conf.urls</tt> instead of
<tt class="docutils literal">django.conf.urls.defaults</tt> to access <tt class="docutils literal">include()</tt>, <tt class="docutils literal">patterns()</tt>, and
<tt class="docutils literal">url()</tt>. The Django team had recently moved these functions and updated the
docs and tutorial to stop using the frowned upon <tt class="docutils literal">from
django.conf.urls.defaults import *</tt>.</li>
<li><strong>Enable the new clickjacking protection</strong>. A nice new feature is some new
middleware that adds the <tt class="docutils literal"><span class="pre">X-Frame-Options</span></tt> header to all response headers.
This provides <a class="reference external" href="http://en.wikipedia.org/wiki/Clickjacking">clickjacking</a> protection in modern browsers.</li>
<li><strong>Add an admin password reset feature</strong>. By adding a few new lines to your
<tt class="docutils literal">urlconf</tt> you get a nifty new password reset feature for your admin.</li>
<li><strong>Update to the new manage.py</strong>. This was the biggest change with the most
impact. The Django team has finally removed a long standing wart with its
<tt class="docutils literal">manage.py</tt> utility. Previously, <tt class="docutils literal">manage.py</tt> used to play games with your
<tt class="docutils literal">PYTHONPATH</tt> which led to confusion when migrating to production. It could
also lead to having your settings imported twice. See the next section in
this blog entry for more on what I did here.</li>
</ol>
</div>
<div class="section" id="reorganizing-for-the-new-manage-py">
<h3>Reorganizing for the new manage.py</h3>
<p>The change with the largest impact for me was reorganizing my directory
structure for the new <tt class="docutils literal">manage.py</tt> command. Before this change, I had organized
my directory structure like this:</p>
<pre class="literal-block">
mysite/
   media/
   static/
   mysite/
      myapp1/
         __init__.py
         models.py
         views.py
         urls.py
      myapp2/
         __init__.py
         models.py
         views.py
         urls.py
      settings/
         __init__.py
         base.py
         local.py
         production.py
         test.py
      apache/
         myproject.wsgi
      logs/
      templates/
      manage.py
      urls.py
   LICENSE
   fabfile.py
   requirements.txt
</pre>
<p>After replacing the contents of my old <tt class="docutils literal">manage.py</tt> with the new content, I
then reorganized my directory structure to this:</p>
<pre class="literal-block">
mysite/
   media/
   static/
   myapp1/
      __init__.py
      models.py
      views.py
      urls.py
   myapp2/
      __init__.py
      models.py
      views.py
      urls.py
   myproject/
      settings/
         __init__.py
         base.py
         local.py
         production.py
         test.py
      apache/
         myproject.wsgi
      logs/
      templates/
      urls.py
   LICENSE
   fabfile.py
   manage.py
   requirements.txt
</pre>
<p>It is a subtle change, but I like it. It now makes it clear that my project is
just an application itself, consisting of the top-level <tt class="docutils literal">urls.py</tt>, settings,
templates and logs. The <tt class="docutils literal">manage.py</tt> file is now at the top level directory
also, which seems right.</p>
<p>I had always made my imports as <tt class="docutils literal">from app.models import MyModel</tt> instead of
<tt class="docutils literal">from myproject.app.models</tt>, so I didn't have to update any imports.</p>
<p>Since I use the &quot;settings as a package&quot; scheme, I did have to update the imports
in my settings files. For example, in my <tt class="docutils literal">local.py</tt> I had to change <tt class="docutils literal">from
settings.base import *</tt> to <tt class="docutils literal">myproject.settings.base import *</tt>.</p>
</div>
<div class="section" id="what-i-didn-t-do">
<h3>What I didn't do</h3>
<p>Django 1.4's largest new feature is probably its support for timezones. I
decided for this project not to take advantage of that. It would require a lot
of changes, and it isn't really worth it for this small site. I may use it on
the next site I convert to Django 1.4, and I will definitely be using it on new
projects.</p>
</div>
<div class="section" id="conclusion">
<h3>Conclusion</h3>
<p>The upgrade process went smoother and quicker than I thought thanks to the
excellent release notes and the Django team's use of Python warnings to flag
deprecated features.</p>
</div>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Installing omniORBpy in a virtualenv]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2012/03/27/installing-omniorbpy-in-a-virtualenv" />
    <id>http://deathofagremmie.com/2012/03/27/installing-omniorbpy-in-a-virtualenv</id>
    <updated>2012-03-27T20:30:00Z</updated>
    <published>2012-03-27T20:30:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Python" />
    <category scheme="http://deathofagremmie.com" term="virtualenv" />
    <category scheme="http://deathofagremmie.com" term="omniORB" />
    <summary type="html"><![CDATA[Installing omniORBpy in a virtualenv]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2012/03/27/installing-omniorbpy-in-a-virtualenv"><![CDATA[<div class="document">
<p>I came across an interesting question on <a class="reference external" href="http://stackoverflow.com/">Stackoverflow</a>, which was essentially
<a class="reference external" href="http://stackoverflow.com/questions/9716611/install-omniorb-python-in-a-virtualenv">How to install omniORBpy in a virtualenv</a>? I've had a lot of experience with
<a class="reference external" href="http://omniorb.sourceforge.net/">omniORB</a>, a high quality, open-source CORBA ORB, at work. omniORB is a C++ ORB,
but it also includes omniORBpy, a thin <a class="reference external" href="http://www.python.org/">Python</a> wrapper around the C++ core.
Despite using omniORBpy extensively, I had never used it in a <a class="reference external" href="http://www.virtualenv.org/">virtualenv</a>.</p>
<p>So I got curious, and gave it a shot. You can read <a class="reference external" href="http://stackoverflow.com/a/9881882/63485">my answer</a>, which the
questioner accepted. I'd love to hear of any better ways of doing it. I worked
it out for Ubuntu, but a Windows solution would be nice to see also.</p>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Upgrading Trac on Windows Gotcha - Part 2]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2012/03/21/upgrading-trac-on-windows-gotcha-part-2" />
    <id>http://deathofagremmie.com/2012/03/21/upgrading-trac-on-windows-gotcha-part-2</id>
    <updated>2012-03-21T20:10:00Z</updated>
    <published>2012-03-21T20:10:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Python" />
    <category scheme="http://deathofagremmie.com" term="Subversion" />
    <category scheme="http://deathofagremmie.com" term="Windows" />
    <category scheme="http://deathofagremmie.com" term="Trac" />
    <summary type="html"><![CDATA[Upgrading Trac on Windows Gotcha - Part 2]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2012/03/21/upgrading-trac-on-windows-gotcha-part-2"><![CDATA[<div class="document">
<p>I have <a class="reference external" href="/2011/09/12/upgrading-trac-on-windows-gotchas">previously reported</a> on some problems I had when upgrading our <a class="reference external" href="http://trac.edgewall.org/">Trac</a>
install at work. Today I attempted another upgrade to <a class="reference external" href="http://subversion.apache.org/">Subversion</a> 1.7.4 and Trac
0.12.3 on Windows. I upgraded to <a class="reference external" href="http://www.python.org/">Python</a> 2.7.2 along the way. I ran into another
problem with the Python bindings to Subversion which took a while to figure out.
I had no problems upgrading Subversion, but Trac could not see our repository.
The symptoms were that the Trac &quot;timeline&quot; and &quot;browse source&quot; features were
missing.</p>
<p>Following the <a class="reference external" href="http://trac.edgewall.org/wiki/TracSubversion#Checklist">Trac troubleshooting advice</a>, I opened an interactive Python
session and tried this:</p>
<pre class="literal-block">
E:\Trac_Data&gt;python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; from svn import client
Traceback (most recent call last):
 File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
 File &quot;C:\Python27\lib\site-packages\svn\client.py&quot;, line 26, in &lt;module&gt;
   from libsvn.client import *
 File &quot;C:\Python27\lib\site-packages\libsvn\client.py&quot;, line 25, in &lt;module&gt;
   _client = swig_import_helper()
 File &quot;C:\Python27\lib\site-packages\libsvn\client.py&quot;, line 21, in swig_import
_helper
   _mod = imp.load_module('_client', fp, pathname, description)
ImportError: DLL load failed: The operating system cannot run %1
</pre>
<p>After some head scratching and googling I finally found the problem. I had used
the Windows .msi installer, <a class="reference external" href="http://sourceforge.net/projects/win32svn/">graciously provided by Alagazam</a>, aka David Darj,
to install Subversion.  This placed the Subversion binaries and DLL's in
<tt class="docutils literal"><span class="pre">C:\Program</span> Files\Subversion\bin</tt>. I then unzipped the Python 2.7 bindings to
the <tt class="docutils literal"><span class="pre">C:\Python27\Lib\site-packages</span></tt> folder. The bindings depend on the DLL's
in the <tt class="docutils literal">Subversion\bin</tt> folder. But unfortunately for me, there were already
two older versions of the DLL's, <tt class="docutils literal">libeay32.dll</tt> and <tt class="docutils literal">ssleay32.dll</tt> on my
path. So when the bindings went looking for those two DLL's, instead of finding
them in <tt class="docutils literal">Subversion\bin</tt>, it found the older versions somewhere else.</p>
<p>To fix this, you can either rearrange your path, or copy those two DLL's to your
<tt class="docutils literal"><span class="pre">Python27\Lib\site-packages\libsvn</span></tt> folder. In the future, I am going to just
copy all the DLL's from <tt class="docutils literal">Subversion\bin</tt> to the <tt class="docutils literal">libsvn</tt> folder.</p>
<p>I examined the pre-built Subversion packages from <a class="reference external" href="http://bitnami.org/">Bitnami</a> and <a class="reference external" href="http://www.collab.net/">CollabNet</a>. They
had packaged all of the Subversion DLL's with the Python bindings together in
the same directory, so this seems reasonable. Later, on the Subversion users'
mailing list, Alagazam gave the nod to this approach.</p>
<p>A big thank you to Alagazam for the help and for the Windows binaries. And of
course thanks to the <a class="reference external" href="http://subversion.apache.org/">Apache</a>, <a class="reference external" href="http://subversion.apache.org/">Subversion</a>, <a class="reference external" href="http://trac.edgewall.org/">Trac</a>, &amp; <a class="reference external" href="http://www.python.org/">Python</a> teams for making
great tools.</p>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[A TeamSpeak 3 viewer with Python & Javascript]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2012/01/20/a-teamspeak-3-viewer-with-python-javascript" />
    <id>http://deathofagremmie.com/2012/01/20/a-teamspeak-3-viewer-with-python-javascript</id>
    <updated>2012-01-20T19:15:00Z</updated>
    <published>2012-01-20T19:15:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Python" />
    <category scheme="http://deathofagremmie.com" term="Javascript" />
    <category scheme="http://deathofagremmie.com" term="TeamSpeak" />
    <summary type="html"><![CDATA[A TeamSpeak 3 viewer with Python & Javascript]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2012/01/20/a-teamspeak-3-viewer-with-python-javascript"><![CDATA[<div class="document">
<div class="section" id="the-problem">
<h3>The Problem</h3>
<p>My gaming clan started using <a class="reference external" href="http://teamspeak.com/?page=teamspeak3">TeamSpeak 3</a> (TS3) for voice communications, so
it wasn't long before we wanted to see who was on the TS3 server from the clan's
server status page. Long ago, before I met Python, I had built the clan a server
status page in PHP. This consisted of cobbling together various home-made and
3rd party PHP scripts for querying game servers (Call of Duty, Battlefield) and
voice servers (TeamSpeak 2 and Mumble). But TeamSpeak 3 was a new one for us,
and I didn't have anything to query that. My interests in PHP are long behind
me, but we needed to add a TS3 viewer to the PHP page. The gaming clan's web
hosting is pretty vanilla; in other words PHP is the first class citizen. If I
really wanted to host a Python app I probably could have resorted to Fast CGI or
something. But I had no experience in that and no desire to go that way.</p>
<p>I briefly thought about finding a 3rd party PHP library to query a TS3 server.
The libraries are out there, but they are as you might expect: overly
complicated and/or pretty amateurish (no public source code repository). I even
considered writing my own PHP code to do the job, so I started looking for any
documentation on the TS3 server query protocol. Luckily, there is a <a class="reference external" href="http://media.teamspeak.com/ts3_literature/TeamSpeak%203%20Server%20Query%20Manual.pdf">TS3
query protocol document</a>, and it is fairly decent.</p>
<p>But, I just could not bring myself to write PHP again. On top of this, the
gaming clan's shared hosting blocks non-standard ports. If I did have a PHP
solution, the outgoing query to the TS3 server would have been blocked by the
host's firewall. It is a hassle to contact their technical support and try to
find a person who knows what a port is and get it unblocked (we've had to do
this over and over as each game comes out). Thus it ultimately boiled down to me
wanting to do this in Python. For me, life is too short to write PHP scripts.</p>
<p>I started thinking about writing a query application in Python using my
dedicated server that I use to host a few <a class="reference external" href="https://www.djangoproject.org">Django</a> powered websites. At first I
thought I'd generate the server status HTML on my server and display it in an
<tt class="docutils literal">&lt;iframe&gt;</tt> on the gaming clan's server. But then it hit me that all I really
needed to do is have my <a class="reference external" href="https://www.djangoproject.org">Django</a> application output a representation of the TS3
server status in <a class="reference external" href="http://json.org">JSON</a>, and then perhaps I could find a slick <a class="reference external" href="http://jquery.org">jQuery</a> tree menu
to display the status graphically. I really liked this idea, so here is a post
about the twists and turns I took implementing it.</p>
</div>
<div class="section" id="the-javascript">
<h3>The Javascript</h3>
<p>My searching turned up several jQuery tree menu plugins, but in the end I
settled on <a class="reference external" href="http://code.google.com/p/dynatree/">dynatree</a>. Dynatree had clear documentation I could understand, it
seems to be actively maintained, and it can generate a menu from JSON. After one
evening of reading the docs, I built a static test HTML page that could display
a tree menu built from JSON. Here the Javascript code I put in the test page's
<tt class="docutils literal">&lt;head&gt;</tt> section:</p>
<div class="highlight"><pre><span class="kd">var</span> <span class="nx">ts3_data</span> <span class="o">=</span> <span class="p">[</span>
   <span class="p">{</span><span class="nx">title</span><span class="o">:</span> <span class="s2">&quot;Phantom Aces&quot;</span><span class="p">,</span> <span class="nx">isFolder</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">expand</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
      <span class="nx">children</span><span class="o">:</span> <span class="p">[</span>
         <span class="p">{</span><span class="nx">title</span><span class="o">:</span> <span class="s2">&quot;MW3&quot;</span><span class="p">,</span> <span class="nx">isFolder</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">expand</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
            <span class="nx">children</span><span class="o">:</span> <span class="p">[</span>
               <span class="p">{</span><span class="nx">title</span><span class="o">:</span> <span class="s2">&quot;Hogan&quot;</span><span class="p">,</span> <span class="nx">icon</span><span class="o">:</span> <span class="s2">&quot;client.png&quot;</span><span class="p">},</span>
               <span class="p">{</span><span class="nx">title</span><span class="o">:</span> <span class="s2">&quot;Fritz!!&quot;</span><span class="p">,</span> <span class="nx">icon</span><span class="o">:</span> <span class="s2">&quot;client.png&quot;</span><span class="p">}</span>
            <span class="p">]</span>
         <span class="p">},</span>
         <span class="p">{</span><span class="nx">title</span><span class="o">:</span> <span class="s2">&quot;COD4&quot;</span><span class="p">,</span> <span class="nx">isFolder</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">expand</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
            <span class="nx">children</span><span class="o">:</span> <span class="p">[</span>
               <span class="p">{</span><span class="nx">title</span><span class="o">:</span> <span class="s2">&quot;Klink&quot;</span><span class="p">,</span> <span class="nx">icon</span><span class="o">:</span> <span class="s2">&quot;client.png&quot;</span><span class="p">}</span>
            <span class="p">]</span>
         <span class="p">},</span>
         <span class="p">{</span><span class="nx">title</span><span class="o">:</span> <span class="s2">&quot;Away&quot;</span><span class="p">,</span> <span class="nx">isFolder</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">children</span><span class="o">:</span> <span class="p">[],</span> <span class="nx">expand</span><span class="o">:</span> <span class="kc">true</span><span class="p">}</span>
     <span class="p">]</span>
   <span class="p">}</span>
<span class="p">];</span>

<span class="nx">$</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
   <span class="nx">$</span><span class="p">(</span><span class="s2">&quot;#ts3-tree&quot;</span><span class="p">).</span><span class="nx">dynatree</span><span class="p">({</span>
      <span class="nx">persist</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span>
      <span class="nx">children</span><span class="o">:</span> <span class="nx">ts3_data</span>
   <span class="p">});</span>
 <span class="p">});</span>
</pre></div>
<p>Note that <tt class="docutils literal">client.png</tt> is a small icon I found that I use in place of
dynatree's default file icon to represent TS3 clients. If I omitted the icon
attribute, the TS3 client would have appeared as a small file icon. Channels
appear as folder icons, and this didn't seem to unreasonable to me. In other
words I had no idea what a channel icon would look like. A folder was fine.</p>
<p>With dynatree, you don't need a lot of HTML markup, it does all the heavy
lifting. You simply have to give it an empty <tt class="docutils literal">&lt;div&gt;</tt> tag it can render
into.</p>
<div class="highlight"><pre><span class="nt">&lt;body&gt;</span>
   <span class="nt">&lt;div</span> <span class="na">id=</span><span class="s">&quot;ts3-tree&quot;</span><span class="nt">&gt;&lt;/div&gt;</span>
<span class="nt">&lt;/body&gt;</span>
<span class="nt">&lt;/html&gt;</span>
</pre></div>
<p>Here is a screenshot of the static test page in action.</p>
<img alt="/images/011-tree1.png" src="/images/011-tree1.png" />
<p>Nice! Thanks dynatree! Now all I need to do is figure out how to dynamically
generate the JSON data and get it into the gaming clan's server status page.</p>
</div>
<div class="section" id="the-python">
<h3>The Python</h3>
<p>Looking through the <a class="reference external" href="http://media.teamspeak.com/ts3_literature/TeamSpeak%203%20Server%20Query%20Manual.pdf">TS3 protocol documentation</a> I was somewhat surprised to
see that TS3 used the Telnet protocol for queries. So from my trusty shell I
telnet'ed into the TS3 server and played with the available commands. I made
notes on what commands I needed to issue to build my status display.</p>
<p>My experiments worked, and I could see a path forward, but there were still some
kinks to be worked out with the TS3 protocol. The data it sent back was escaped
in a strange way for one thing. I would have to post-process the data in Python
before I could use it. I didn't want to reinvent the wheel, so I did a quick
search for Python libraries for working with TS3. I found a few, but quickly
settled on Andrew William's <a class="reference external" href="http://pypi.python.org/pypi/python-ts3/0.1">python-ts3</a> library. It was small, easy to
understand, had tests, and a GitHub page. Perfect.</p>
<p>One of the great things about Python, of course, is the interactive shell. Armed
with the <a class="reference external" href="http://media.teamspeak.com/ts3_literature/TeamSpeak%203%20Server%20Query%20Manual.pdf">TS3 protocol documentation</a>, <a class="reference external" href="http://pypi.python.org/pypi/python-ts3/0.1">python-ts3</a>, and the Python shell, I was
able to interactively connect to the TS3 server and poke around again. This time
I was sitting above telnet using <a class="reference external" href="http://pypi.python.org/pypi/python-ts3/0.1">python-ts3</a> and I confirmed it would do the job
for me.</p>
<p>Another evening was spent coding up a Django view to query the TS3 server using
<a class="reference external" href="http://pypi.python.org/pypi/python-ts3/0.1">python-ts3</a> and to output the channel status as JSON.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">django.core.cache</span> <span class="kn">import</span> <span class="n">cache</span>
<span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">HttpResponse</span><span class="p">,</span> <span class="n">HttpResponseServerError</span>
<span class="kn">from</span> <span class="nn">django.utils</span> <span class="kn">import</span> <span class="n">simplejson</span>
<span class="kn">import</span> <span class="nn">ts3</span>

<span class="n">CACHE_KEY</span> <span class="o">=</span> <span class="s">&#39;ts3-json&#39;</span>
<span class="n">CACHE_TIMEOUT</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="mi">60</span>

<span class="k">def</span> <span class="nf">ts3_query</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    Query the TeamSpeak3 server for status, and output a JSON</span>
<span class="sd">    representation.</span>

<span class="sd">    The JSON we return is targeted towards the jQuery plugin Dynatree</span>
<span class="sd">    http://code.google.com/p/dynatree/</span>

<span class="sd">    &quot;&quot;&quot;</span>
    <span class="c"># Do we have the result cached?</span>
    <span class="n">result</span> <span class="o">=</span> <span class="n">cache</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">CACHE_KEY</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">result</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">content_type</span><span class="o">=</span><span class="s">&#39;application/json&#39;</span><span class="p">)</span>

    <span class="c"># Cache miss, go query the remote server</span>

    <span class="k">try</span><span class="p">:</span>
        <span class="n">svr</span> <span class="o">=</span> <span class="n">ts3</span><span class="o">.</span><span class="n">TS3Server</span><span class="p">(</span><span class="n">settings</span><span class="o">.</span><span class="n">TS3_IP</span><span class="p">,</span> <span class="n">settings</span><span class="o">.</span><span class="n">TS3_PORT</span><span class="p">,</span>
                <span class="n">settings</span><span class="o">.</span><span class="n">TS3_VID</span><span class="p">)</span>
    <span class="k">except</span> <span class="n">ts3</span><span class="o">.</span><span class="n">ConnectionError</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">HttpResponseServerError</span><span class="p">()</span>

    <span class="n">response</span> <span class="o">=</span> <span class="n">svr</span><span class="o">.</span><span class="n">send_command</span><span class="p">(</span><span class="s">&#39;serverinfo&#39;</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">response</span><span class="p">[</span><span class="s">&#39;msg&#39;</span><span class="p">]</span> <span class="o">!=</span> <span class="s">&#39;ok&#39;</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">HttpResponseServerError</span><span class="p">()</span>
    <span class="n">svr_info</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>

    <span class="n">response</span> <span class="o">=</span> <span class="n">svr</span><span class="o">.</span><span class="n">send_command</span><span class="p">(</span><span class="s">&#39;channellist&#39;</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">response</span><span class="p">[</span><span class="s">&#39;msg&#39;</span><span class="p">]</span> <span class="o">!=</span> <span class="s">&#39;ok&#39;</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">HttpResponseServerError</span><span class="p">()</span>
    <span class="n">channel_list</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">data</span>

    <span class="n">response</span> <span class="o">=</span> <span class="n">svr</span><span class="o">.</span><span class="n">send_command</span><span class="p">(</span><span class="s">&#39;clientlist&#39;</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">response</span><span class="p">[</span><span class="s">&#39;msg&#39;</span><span class="p">]</span> <span class="o">!=</span> <span class="s">&#39;ok&#39;</span><span class="p">:</span>
        <span class="k">return</span> <span class="n">HttpResponseServerError</span><span class="p">()</span>
    <span class="n">client_list</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">data</span>

    <span class="c"># Start building the channel / client tree.</span>
    <span class="c"># We save tree nodes in a dictionary, keyed by their id so we can find</span>
    <span class="c"># them later in order to support arbitrary channel hierarchies.</span>
    <span class="n">channels</span> <span class="o">=</span> <span class="p">{}</span>

    <span class="c"># Build the root, or channel 0</span>
    <span class="n">channels</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
        <span class="s">&#39;title&#39;</span><span class="p">:</span> <span class="n">svr_info</span><span class="p">[</span><span class="s">&#39;virtualserver_name&#39;</span><span class="p">],</span>
        <span class="s">&#39;isFolder&#39;</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span>
        <span class="s">&#39;expand&#39;</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span>
        <span class="s">&#39;children&#39;</span><span class="p">:</span> <span class="p">[]</span>
    <span class="p">}</span>

    <span class="c"># Add the channels to our tree</span>

    <span class="k">for</span> <span class="n">channel</span> <span class="ow">in</span> <span class="n">channel_list</span><span class="p">:</span>
        <span class="n">node</span> <span class="o">=</span> <span class="p">{</span>
            <span class="s">&#39;title&#39;</span><span class="p">:</span> <span class="n">channel</span><span class="p">[</span><span class="s">&#39;channel_name&#39;</span><span class="p">],</span>
            <span class="s">&#39;isFolder&#39;</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span>
            <span class="s">&#39;expand&#39;</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span>
            <span class="s">&#39;children&#39;</span><span class="p">:</span> <span class="p">[]</span>
        <span class="p">}</span>
        <span class="n">parent</span> <span class="o">=</span> <span class="n">channels</span><span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">channel</span><span class="p">[</span><span class="s">&#39;pid&#39;</span><span class="p">])]</span>
        <span class="n">parent</span><span class="p">[</span><span class="s">&#39;children&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
        <span class="n">channels</span><span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">channel</span><span class="p">[</span><span class="s">&#39;cid&#39;</span><span class="p">])]</span> <span class="o">=</span> <span class="n">node</span>

    <span class="c"># Add the clients to the tree</span>

    <span class="k">for</span> <span class="n">client</span> <span class="ow">in</span> <span class="n">client_list</span><span class="p">:</span>
        <span class="k">if</span> <span class="n">client</span><span class="p">[</span><span class="s">&#39;client_type&#39;</span><span class="p">]</span> <span class="o">==</span> <span class="s">&#39;0&#39;</span><span class="p">:</span>
            <span class="n">node</span> <span class="o">=</span> <span class="p">{</span>
                <span class="s">&#39;title&#39;</span><span class="p">:</span> <span class="n">client</span><span class="p">[</span><span class="s">&#39;client_nickname&#39;</span><span class="p">],</span>
                <span class="s">&#39;icon&#39;</span><span class="p">:</span> <span class="s">&#39;client.png&#39;</span>
            <span class="p">}</span>
            <span class="n">channel</span> <span class="o">=</span> <span class="n">channels</span><span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">client</span><span class="p">[</span><span class="s">&#39;cid&#39;</span><span class="p">])]</span>
            <span class="n">channel</span><span class="p">[</span><span class="s">&#39;children&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>

    <span class="n">tree</span> <span class="o">=</span> <span class="p">[</span><span class="n">channels</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span>

    <span class="c"># convert to JSON</span>
    <span class="n">json</span> <span class="o">=</span> <span class="n">simplejson</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>

    <span class="n">cache</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">CACHE_KEY</span><span class="p">,</span> <span class="n">json</span><span class="p">,</span> <span class="n">CACHE_TIMEOUT</span><span class="p">)</span>

    <span class="k">return</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="n">json</span><span class="p">,</span> <span class="n">content_type</span><span class="o">=</span><span class="s">&#39;application/json&#39;</span><span class="p">)</span>
</pre></div>
<p>I have to make three queries to the TS3 server to get all the information I
need. The <tt class="docutils literal">serverinfo</tt> command is issued to retrieve the TS3 virtual server's
name. The <tt class="docutils literal">channellist</tt> command retrieves the list of channels. The
<tt class="docutils literal">clientlist</tt> command gets the list of TS3 clients that are currently
connected. For more information on these three commands see the TS3 query
protocol document.</p>
<p>The only real tricky part of this code was figuring out how to represent an
arbitrary, deeply-nested channel tree in Python. I ended up guessing that
<tt class="docutils literal">cid</tt> meant channel ID and <tt class="docutils literal">pid</tt> meant parent ID in the TS3 query data. I
squirrel away the channels in a <tt class="docutils literal">channels</tt> dictionary, keyed by channel ID.
The root channel has an ID of 0. While iterating over the channel list, I can
retrieve the parent channel from the <tt class="docutils literal">channels</tt> dictionary by ID and append
the new channel to the parent's <tt class="docutils literal">children</tt> list. Clients are handled the same
way, but have different attributes. By inspecting the <tt class="docutils literal">clientlist</tt> data in the
Python shell, I noticed that my Telnet client also showed up in that list.
However it had a <tt class="docutils literal">client_type</tt> of 1, whereas the normal PC clients had a
<tt class="docutils literal">client_type</tt> of 0.</p>
<p>I decided to cache the results for 2 minutes to reduce hits on the TS3 server,
as it has flood protection. This probably isn't needed given the size of our
gaming clan, but Django makes it easy to do, so why not?</p>
</div>
<div class="section" id="putting-it-all-together">
<h3>Putting it all together</h3>
<p>At this point I knew how to use my Django application to query the TS3 server
and build status in JSON format. I also knew what the Javascript and HTML on the
gaming clan's server status page (written in PHP) had to look like to render
that JSON status.</p>
<p>The problem was the server status page was on one server, and my Django
application was on another. At first I thought it would be no problem for the
Javascript to do a <tt class="docutils literal">GET</tt> on my Django server and retrieve the JSON. However I
had some vague memory of the browser security model, and after some googling I
was reminded of the <a class="reference external" href="http://en.wikipedia.org/wiki/Same_origin_policy">same origin policy</a>. Rats. That wasn't going to work.</p>
<p>I briefly researched <a class="reference external" href="http://en.wikipedia.org/wiki/JSONP">JSONP</a>, which is the technique that Facebook &amp; Google use
to embed those little &quot;like&quot; and &quot;+1&quot; buttons on your web pages. But in the end
it was just as easy to have the PHP script make the <tt class="docutils literal">GET</tt> request to my Django
application using a <a class="reference external" href="http://php.net/manual/en/function.file-get-contents.php">file_get_contents()</a> call. The PHP can then embed the JSON
directly into the server status page:</p>
<div class="highlight"><pre><span class="x">$ts3_source = &#39;http://example.com/ts3/&#39;;</span>
<span class="x">$ts3_json = file_get_contents($ts3_source);</span>

<span class="x">require_once &#39;header.php&#39;;</span>
</pre></div>
<p>And in header.php, some HTML sprinkled with some PHP:</p>
<div class="highlight"><pre><span class="nt">&lt;script </span><span class="na">type=</span><span class="s">&quot;text/javascript&quot;</span><span class="nt">&gt;</span>
   <span class="kd">var</span> <span class="nx">ts3_data</span> <span class="o">=</span> <span class="o">&lt;?</span><span class="nx">php</span> <span class="nx">echo</span> <span class="nx">$ts3_json</span><span class="p">;</span> <span class="o">?&gt;</span><span class="p">;</span>

   <span class="nx">$</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
      <span class="nx">$</span><span class="p">(</span><span class="s2">&quot;#ts3-tree&quot;</span><span class="p">).</span><span class="nx">dynatree</span><span class="p">({</span>
         <span class="nx">persist</span><span class="o">:</span> <span class="kc">false</span><span class="p">,</span>
         <span class="nx">children</span><span class="o">:</span> <span class="nx">ts3_data</span>
      <span class="p">});</span>
    <span class="p">});</span>
<span class="nt">&lt;/script&gt;</span>
</pre></div>
<p>That did the trick. In the end I had to touch a little PHP, but it was
tolerable.  That was a very round-about solution to building a TS3 viewer in
Python and Javascript. While I doubt you will have the same strange requirements
that I had (multiple servers), I hope you can see how to combine a few
technologies to make a TS3 viewer in Python.</p>
</div>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Who's Online with Redis & Python, a slight return]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2011/12/17/who-s-online-with-redis-python-a-slight-return" />
    <id>http://deathofagremmie.com/2011/12/17/who-s-online-with-redis-python-a-slight-return</id>
    <updated>2011-12-17T19:05:00Z</updated>
    <published>2011-12-17T19:05:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Python" />
    <category scheme="http://deathofagremmie.com" term="Redis" />
    <summary type="html"><![CDATA[Who's Online with Redis & Python, a slight return]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2011/12/17/who-s-online-with-redis-python-a-slight-return"><![CDATA[<div class="document">
<p>In a <a class="reference external" href="http://deathofagremmie.com/2011/04/25/a-better-who-s-online-with-redis-python/">previous post</a>, I blogged about building a &quot;Who's Online&quot; feature using
<a class="reference external" href="http://redis.io/">Redis</a> and <a class="reference external" href="http://www.python.org">Python</a> with <a class="reference external" href="https://github.com/andymccurdy/redis-py">redis-py</a>.  I've been integrating <a class="reference external" href="http://celeryproject.org">Celery</a> into my
website, and I stumbled across this old code. Since I made that post, I
discovered yet another cool feature in Redis: sorted sets. So here is an even
better way of implementing this feature using Redis sorted sets.</p>
<p>A sorted set in Redis is like a regular set, but each member has a numeric
score. When you add a member to a sorted set, you also specify the score for
that member. You can then retrieve set members if their score falls into a
certain range. You can also easily remove members outside a given score range.</p>
<p>For a &quot;Who's Online&quot; feature, we need a sorted set to represent the set
of all users online. Whenever we see a user, we insert that user into the set
along with the current time as their score. This is accomplished with the Redis
<a class="reference external" href="http://redis.io/commands/zadd">zadd</a> command.  If the user is already in the set, <a class="reference external" href="http://redis.io/commands/zadd">zadd</a> simply updates
their score with the current time.</p>
<p>To obtain the curret list of who's online, we use the <a class="reference external" href="http://redis.io/commands/zrangebyscore">zrangebyscore</a> command to
retrieve the list of users who's score (time) lies between, say, 15 minutes ago,
until now.</p>
<p>Periodically, we need to remove stale members from the set. This can be
accomplished by using the <a class="reference external" href="http://redis.io/commands/zremrangebyscore">zremrangebyscore</a> command. This command will remove
all members that have a score between minimum and maximum values. In this case,
we can use the beginning of time for the minimum, and 15 minutes ago for the
maximum.</p>
<p>That's really it in a nutshell. This is much simpler than my previous
solution which used two sets.</p>
<p>So let's look at some code. The first problem we need to solve is how to
convert a Python <tt class="docutils literal">datetime</tt> object into a score. This can be accomplished by
converting the <tt class="docutils literal">datetime</tt> into a POSIX timestamp integer, which is the number
of seconds from the UNIX epoch of January 1, 1970.</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">datetime</span>
<span class="kn">import</span> <span class="nn">time</span>

<span class="k">def</span> <span class="nf">to_timestamp</span><span class="p">(</span><span class="n">dt</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    Turn the supplied datetime object into a UNIX timestamp integer.</span>

<span class="sd">    &quot;&quot;&quot;</span>
    <span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">mktime</span><span class="p">(</span><span class="n">dt</span><span class="o">.</span><span class="n">timetuple</span><span class="p">()))</span>
</pre></div>
<p>With that handy function, here are some examples of the operations described
above.</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">redis</span>

<span class="c"># Redis set keys:</span>
<span class="n">USER_SET_KEY</span> <span class="o">=</span> <span class="s">&quot;whos_online:users&quot;</span>

<span class="c"># the period over which we collect who&#39;s online stats:</span>
<span class="n">MAX_AGE</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">minutes</span><span class="o">=</span><span class="mi">15</span><span class="p">)</span>

<span class="c"># obtain a connection to redis:</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">redis</span><span class="o">.</span><span class="n">StrictRedis</span><span class="p">()</span>

<span class="c"># add/update a user to the who&#39;s online set:</span>

<span class="n">username</span> <span class="o">=</span> <span class="s">&quot;sally&quot;</span>
<span class="n">ts</span> <span class="o">=</span> <span class="n">to_timestamp</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
<span class="n">conn</span><span class="o">.</span><span class="n">zadd</span><span class="p">(</span><span class="n">USER_SET_KEY</span><span class="p">,</span> <span class="n">ts</span><span class="p">,</span> <span class="n">username</span><span class="p">)</span>

<span class="c"># retrieve the list of users who have been active in the last MAX_AGE minutes</span>

<span class="n">now</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="nb">min</span> <span class="o">=</span> <span class="n">to_timestamp</span><span class="p">(</span><span class="n">now</span> <span class="o">-</span> <span class="n">MAX_AGE</span><span class="p">)</span>
<span class="nb">max</span> <span class="o">=</span> <span class="n">to_timestamp</span><span class="p">(</span><span class="n">now</span><span class="p">)</span>

<span class="n">whos_online</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">zrangebyscore</span><span class="p">(</span><span class="n">USER_SET_KEY</span><span class="p">,</span> <span class="nb">min</span><span class="p">,</span> <span class="nb">max</span><span class="p">)</span>

<span class="c"># e.g. whos_online = [&#39;sally&#39;, &#39;harry&#39;, &#39;joe&#39;]</span>

<span class="c"># periodically remove stale members</span>

<span class="n">cutoff</span> <span class="o">=</span> <span class="n">to_timestamp</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">MAX_AGE</span><span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">zremrangebyscore</span><span class="p">(</span><span class="n">USER_SET_KEY</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">cutoff</span><span class="p">)</span>
</pre></div>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Upgrading Trac on Windows Gotchas]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2011/09/12/upgrading-trac-on-windows-gotchas" />
    <id>http://deathofagremmie.com/2011/09/12/upgrading-trac-on-windows-gotchas</id>
    <updated>2011-09-12T22:15:00Z</updated>
    <published>2011-09-12T22:15:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Python" />
    <category scheme="http://deathofagremmie.com" term="Subversion" />
    <category scheme="http://deathofagremmie.com" term="Windows" />
    <category scheme="http://deathofagremmie.com" term="Trac" />
    <summary type="html"><![CDATA[Upgrading Trac on Windows Gotchas]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2011/09/12/upgrading-trac-on-windows-gotchas"><![CDATA[<div class="document">
<p>At work, we are outfitted with Windows servers. Despite this obstacle, I managed
to install <a class="reference external" href="http://trac.edgewall.org/">Trac</a> and <a class="reference external" href="http://subversion.apache.org/">Subversion</a> a few years ago. During a break in the action,
we decided to update Subversion (SVN) and Trac. Since we are on Windows, this
means we have to rely on the <a class="reference external" href="http://sourceforge.net/projects/win32svn/">kindness of strangers</a> for Subversion binaries. I
ran into a couple of gotchas I'd like to document here to help anyone else who
runs into these.</p>
<p>I managed to get Subversion and Trac up and running without any real problems.
However when Trac needed to access SVN to display changesets or a timeline, for
example, I got an error:</p>
<p><tt class="docutils literal">tracerror: unsupported version control system &quot;svn&quot; no module named _fs</tt></p>
<p>After some googling, I finally found that this issue is <a class="reference external" href="http://trac.edgewall.org/wiki/TracSubversion">documented on the Trac
wiki</a>, but it was kind of hard to find. To fix this problem, you have to rename
the Python SVN binding's DLLs to <tt class="docutils literal">*.pyd</tt>. Specifically, change the
<tt class="docutils literal"><span class="pre">libsvn/*.dll</span></tt> files to <tt class="docutils literal"><span class="pre">libsvn/*.pyd</span></tt>, but don't change the name of
<tt class="docutils literal"><span class="pre">libsvn_swig_py-1.dll</span></tt>. I'd really like to hear an explanation of why one
needs to do this. Why doesn't the Python-Windows build process do this for you?</p>
<p>The second problem I ran into dealt with <a class="reference external" href="http://code.google.com/p/modwsgi/">mod_wsgi</a> on Windows. Originally, a few
years ago, I setup Trac to run under <a class="reference external" href="http://www.modpython.org/">mod_python</a>. mod_python has long been
out of favor, so I decided to cut over to mod_wsgi. On my Linux boxes, I always
run mod_wsgi in daemon mode. When I tried to configure this on Windows, Apache
complained about an unknown directive <tt class="docutils literal">WSGIDaemonProcess</tt>. It turns out that
<a class="reference external" href="http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess">this mode is not supported on Windows</a>. You'll have to use the embedded mode on
Windows.</p>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Implementing OAuth using Google's Python Client Library]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2011/07/04/implementing-oauth-using-google-s-python-client-library" />
    <id>http://deathofagremmie.com/2011/07/04/implementing-oauth-using-google-s-python-client-library</id>
    <updated>2011-07-04T13:00:00Z</updated>
    <published>2011-07-04T13:00:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Python" />
    <category scheme="http://deathofagremmie.com" term="OAuth" />
    <category scheme="http://deathofagremmie.com" term="GData" />
    <category scheme="http://deathofagremmie.com" term="Google" />
    <summary type="html"><![CDATA[Implementing OAuth using Google's Python Client Library]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2011/07/04/implementing-oauth-using-google-s-python-client-library"><![CDATA[<div class="document">
<p>My <a class="reference external" href="http://djangoproject.com">Django</a> powered website allows users to submit events for a site calendar
that is built upon Google Calendar.  After an admin approves events, I use
Google's <a class="reference external" href="http://code.google.com/apis/calendar/data/2.0/developers_guide_python.html">Python Client Library</a> to add, delete, or update events on the Google
calendar associated with my personal Google account. I wrote this application a
few years ago, and it used the <a class="reference external" href="http://code.google.com/apis/calendar/data/2.0/developers_guide_python.html#AuthClientLogin">ClientLogin</a> method for authentication. I
recently decided to upgrade this to the <a class="reference external" href="http://code.google.com/apis/gdata/docs/auth/oauth.html">OAuth</a> authentication method. The
ClientLogin method isn't very secure and it doesn't play well with Google's
<a class="reference external" href="http://googleblog.blogspot.com/2011/02/advanced-sign-in-security-for-your.html">two-step verification</a>. After hearing about a friend who had his GMail account
compromised and all his email deleted I decided it was long past due to get
two-step verification on my account. But first I needed to upgrade my web
application to OAuth.</p>
<p>In this post I'll boil down the code I used to implement the elaborate OAuth
dance. It really isn't that much code, but the Google documentation is somewhat
confusing and scattered across a bewildering number of documents. I found at
least one error in the documentation that I will point out. Although I am using
Django, I will omit details specific to Django where I can.</p>
<p>In addition to switching from ClientLogin to OAuth, I also upgraded to version
2.0 of the Google Data API. This had more implications for my calendar-specific
code, and perhaps I can go over that in a future post.</p>
<div class="section" id="getting-started-and-registering-with-google">
<h3>Getting started and registering with Google</h3>
<p>To understand the basics of OAuth, I suggest you read <a class="reference external" href="http://code.google.com/apis/accounts/docs/OAuth.html">OAuth 1.0 for Web
Applications</a>. I decided to go for maximum security and use RSA-SHA1 signing on
all my requests to Google. This requires that I verify my domain and then
<a class="reference external" href="http://code.google.com/apis/accounts/docs/RegistrationForWebAppsAuto.html">register my application</a> with Google, which includes uploading a security
certificate. Google provides documentation that describes how you can <a class="reference external" href="http://code.google.com/apis/gdata/docs/auth/oauth.html#GeneratingKeyCert">create a
self-signing private key and certificate</a> using OpenSSL.</p>
</div>
<div class="section" id="fetching-a-request-token-and-authorizing-access">
<h3>Fetching a Request Token and authorizing access</h3>
<p>To perform the first part of the OAuth dance, you must ask Google for a request
token. When you make this request, you state the &quot;scope&quot; of your future work by
listing the Google resources you are going to access. In our case, this is the
calendar resources. You also provide a &quot;consumer key&quot; that Google assigned to
you when you registered your application. This allows Google to retrieve the
security certificate you previously uploaded when you registered. This is very
important because this request is going to be signed with your private key.
Fortunately the Python library takes care of all the signing details, you simply
must provide your private key in PEM format. And finally, you provide a
&quot;callback URL&quot; that Google will send your browser to after you (or your users)
have manually authorized this request.</p>
<p>Once you have received the request token from Google, you have to squirrel it
away somewhere, then redirect your (or your user's) browser to a Google
authorization page. Once the user has authorized your application, Google sends
the browser to the callback URL to continue the process. Here I show the
distilled code I used that asks for a request token, then sends the user to the
authorization page.</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">gdata.gauth</span>
<span class="kn">from</span> <span class="nn">gdata.calendar_resource.client</span> <span class="kn">import</span> <span class="n">CalendarResourceClient</span>

<span class="n">USER_AGENT</span> <span class="o">=</span> <span class="s">&#39;mydomain-myapp-v1&#39;</span> <span class="c"># my made up user agent string</span>

<span class="n">client</span> <span class="o">=</span> <span class="n">CalendarResourceClient</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">source</span><span class="o">=</span><span class="n">USER_AGENT</span><span class="p">)</span>

<span class="c"># obtain my private key that I saved previously on the filesystem:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">settings</span><span class="o">.</span><span class="n">GOOGLE_OAUTH_PRIVATE_KEY_PATH</span><span class="p">,</span> <span class="s">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
    <span class="n">rsa_key</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>

<span class="c"># Ask for a request token:</span>
<span class="c"># scopes - a list of scope strings that the request token is for. See</span>
<span class="c"># http://code.google.com/apis/gdata/faq.html#AuthScopes</span>
<span class="c"># callback_url - URL to send the user after authorizing our app</span>

<span class="n">scopes</span> <span class="o">=</span> <span class="p">[</span><span class="s">&#39;https://www.google.com/calendar/feeds/&#39;</span><span class="p">]</span>
<span class="n">callback_url</span> <span class="o">=</span> <span class="s">&#39;http://example.com/some/url/to/callback&#39;</span>

<span class="n">request_token</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">GetOAuthToken</span><span class="p">(</span>
        <span class="n">scopes</span><span class="p">,</span>
        <span class="n">callback_url</span><span class="p">,</span>
        <span class="n">settings</span><span class="o">.</span><span class="n">GOOGLE_OAUTH_CONSUMER_KEY</span><span class="p">,</span> <span class="c"># from the registration process</span>
        <span class="n">rsa_private_key</span><span class="o">=</span><span class="n">rsa_key</span><span class="p">)</span>

<span class="c"># Before redirecting, save the request token somewhere; here I place it in</span>
<span class="c"># the session (this line is Django specific):</span>
<span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="n">REQ_TOKEN_SESSION_KEY</span><span class="p">]</span> <span class="o">=</span> <span class="n">request_token</span>

<span class="c"># Generate the authorization URL.</span>
<span class="c"># Despite the documentation, don&#39;t do this:</span>
<span class="c">#    auth_url = request_token.generate_authorization_url(domain=None)</span>
<span class="c"># Do this instead if you are not using a Google Apps domain:</span>
<span class="n">auth_url</span> <span class="o">=</span> <span class="n">request_token</span><span class="o">.</span><span class="n">generate_authorization_url</span><span class="p">()</span>

<span class="c"># Now redirect the user somehow to the auth_url; here is how you might do</span>
<span class="c"># it in Django:</span>
<span class="k">return</span> <span class="n">HttpResponseRedirect</span><span class="p">(</span><span class="n">auth_url</span><span class="p">)</span>
</pre></div>
<p>A couple of notes on the above:</p>
<ul class="simple">
<li>You don't have to use <tt class="docutils literal">CalendarResourceClient</tt>, it just made the most sense
for me since I am doing calendar stuff later on. Any class that inherits from
<tt class="docutils literal">gdata.client.GDClient</tt> will work. You might be able to use that class
directly. Google uses <tt class="docutils literal">gdata.docs.client.DocsClient</tt> in their examples.</li>
<li>I chose to store my private key in a file rather than the database. If you do
so, it's probably a good idea to make the file readable only to the user your
webserver runs your application as.</li>
<li>After getting the request token you must save it somehow. You can save it in
the session, the database, or perhaps a file. Since this is only temporary, I
chose to save it in the session. The code I have here is Django specific.</li>
<li>When generating the authorization URL, don't pass in <tt class="docutils literal">domain=None</tt> if you
aren't using a Google Apps domain like the documentation states. This appears
to be an error in the documentation. Just omit it and let it use the default
value of <tt class="docutils literal">&quot;default&quot;</tt> (see the source code).</li>
<li>After using the request token to generate the authorization URL, redirect the
browser to it.</li>
</ul>
</div>
<div class="section" id="extracting-and-upgrading-to-an-access-token">
<h3>Extracting and upgrading to an Access Token</h3>
<p>The user will then be taken to a Google authorization page. The page will show the
user what parts of their Google account your application is trying to access
using the information you provided in the <tt class="docutils literal">scopes</tt> parameter. If the user
accepts, Google will then redirect the browser to your callback URL where we can
complete the process.</p>
<p>The code running at our callback URL must retrieve the request token that we
saved earlier, and combine that with certain <tt class="docutils literal">GET</tt> parameters Google attached
to our callback URL. This is all done for us by the Python library. We then send
this new token back to Google to upgrade it to an actual access token. If this
succeeds, we can then save this new access token in our database for use in
subsequent Google API operations. The access token is a Python object, so you
can serialize it use the pickle module, or use routines provided by Google
(shown below).</p>
<div class="highlight"><pre><span class="c"># Code running at our callback URL:</span>
<span class="c"># Retrieve the request token we saved earlier in our session</span>
<span class="n">saved_token</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">session</span><span class="p">[</span><span class="n">REQ_TOKEN_SESSION_KEY</span><span class="p">]</span>

<span class="c"># Authorize it by combining it with GET parameters received from Google</span>
<span class="n">request_token</span> <span class="o">=</span> <span class="n">gdata</span><span class="o">.</span><span class="n">gauth</span><span class="o">.</span><span class="n">AuthorizeRequestToken</span><span class="p">(</span><span class="n">saved_token</span><span class="p">,</span>
                    <span class="n">request</span><span class="o">.</span><span class="n">build_absolute_uri</span><span class="p">())</span>

<span class="c"># Upgrade it to an access token</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">CalendarResourceClient</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">source</span><span class="o">=</span><span class="n">USER_AGENT</span><span class="p">)</span>
<span class="n">access_token</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">GetAccessToken</span><span class="p">(</span><span class="n">request_token</span><span class="p">)</span>

<span class="c"># Now save access_token somewhere, e.g. a database. So first serialize it:</span>
<span class="n">access_token_str</span> <span class="o">=</span>  <span class="n">gdata</span><span class="o">.</span><span class="n">gauth</span><span class="o">.</span><span class="n">TokenToBlob</span><span class="p">(</span><span class="n">access_token</span><span class="p">)</span>

<span class="c"># Save to database (details omitted)</span>
</pre></div>
<p>Some notes on the above code:</p>
<ul class="simple">
<li>Once called back, our code must retrieve the request token we saved in our
session. The code shown is specific to Django.</li>
<li>We then combine this saved request token with certain <tt class="docutils literal">GET</tt> parameters that
Google added to our callback URL. The <tt class="docutils literal">AuthorizeRequestToken</tt> function takes care of
those details for us. The second argument to that function requires the full URL
including <tt class="docutils literal">GET</tt> parameters as a string. Here I populate that argument by
using a Django-specific method of retrieving that information.</li>
<li>Finally, you upgrade your token to an access token by making one last call to
Google. You should now save a serialized version of this access token in your
database for future use.</li>
</ul>
</div>
<div class="section" id="using-your-shiny-new-access-token">
<h3>Using your shiny new Access Token</h3>
<p>Once you have saved your access token, you won't have to do this crazy dance
again until the token either expires, or the user revokes your application's
access to the Google account. To use it in a calendar operation, for example,
you simply retrieve it from your database, deserialize it, and then use it to
create a <tt class="docutils literal">CalendarClient</tt>.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">gdata.calendar.client</span> <span class="kn">import</span> <span class="n">CalendarClient</span>

<span class="c"># retrieve access token from the database:</span>
<span class="n">access_token_str</span> <span class="o">=</span> <span class="o">...</span>
<span class="n">access_token</span> <span class="o">=</span> <span class="n">gdata</span><span class="o">.</span><span class="n">gauth</span><span class="o">.</span><span class="n">TokenFromBlob</span><span class="p">(</span><span class="n">access_token_str</span><span class="p">)</span>

<span class="n">client</span> <span class="o">=</span> <span class="n">CalendarClient</span><span class="p">(</span><span class="n">source</span><span class="o">=</span><span class="n">USER_AGENT</span><span class="p">,</span> <span class="n">auth_token</span><span class="o">=</span><span class="n">access_token</span><span class="p">)</span>

<span class="c"># now use client to make calendar operations...</span>
</pre></div>
</div>
<div class="section" id="conclusion">
<h3>Conclusion</h3>
<p>The main reason I wrote this blog post is I wanted to show a concrete example of
using RSA-SHA1 and version 2.0 of the Google API together. All of the
information I have presented is in the Google documentation, but it is spread
across several documents and jumbled up with example code for version 1.0 and
HMAC-SHA1.  Do not be afraid to look at the source code for the Python client
library.  Despite Google's strange habit of ignoring <a class="reference external" href="http://www.python.org/dev/peps/pep-0008/">PEP-8</a> and using
LongJavaLikeMethodNames, the code is logical and easy to read. Their library is
built up in layers, and you may have to dip down a few levels to find out what
is going on, but it is fairly straightforward to read if you combine it with
their online documentation.</p>
<p>I hope someone finds this useful. Your feedback is welcome.</p>
</div>
</div>
]]></content>
  </entry>
  <entry>
    <author>
      <name>Brian Neal</name>
      <uri>http://deathofagremmie.com</uri>
    </author>
    <title type="html"><![CDATA[Contributing to open source - a success story and advice for newbies]]></title>
    <link rel="alternate" type="text/html" href="http://deathofagremmie.com/2011/06/23/contributing-to-open-source-a-success-story-and-advice-for-newbies" />
    <id>http://deathofagremmie.com/2011/06/23/contributing-to-open-source-a-success-story-and-advice-for-newbies</id>
    <updated>2011-06-23T21:45:00Z</updated>
    <published>2011-06-23T21:45:00Z</published>
    <category scheme="http://deathofagremmie.com" term="Subversion" />
    <category scheme="http://deathofagremmie.com" term="Open Source" />
    <summary type="html"><![CDATA[Contributing to open source - a success story and advice for newbies]]></summary>
    <content type="html" xml:base="http://deathofagremmie.com/2011/06/23/contributing-to-open-source-a-success-story-and-advice-for-newbies"><![CDATA[<div class="document">
<p>Recently, my team at work found a <a class="reference external" href="http://subversion.tigris.org/issues/show_bug.cgi?id=3919">bug in Subversion</a>, I submitted a patch, and it
was accepted!  This was very exciting for me so I thought I would share this
story in the hopes of inspiring others to contribute to open source projects.
It may not be as hard as you might think!</p>
<div class="section" id="the-bug">
<h3>The Bug</h3>
<p>We use <a class="reference external" href="http://subversion.apache.org/">Subversion</a> at work for revision control. My colleague and I were trying
to merge a branch back to trunk when we ran into some strange behavior. We make
use of Subversion properties, which allow you to attach arbitrary metadata to
files and directories. Our project has to deliver our source code and
documentation to the customer in a required directory format (can you guess who
our customer is?). However not all files need to be sent to the customer. To
solve this problem we use a simple &quot;yes/no&quot; delivery property on each file to
control whether it is delivered or not. Before making a delivery, a script is
run that prunes out the files that have the delivery flag set to &quot;no&quot;.</p>
<p>When our merge was completed, many files were marked with having merge conflicts
on the delivery property. Looking through the logs it was discovered that after
we had made our branch, someone had changed the delivery property on some files
to &quot;yes&quot; on the trunk. Someone else had also changed the delivery property
independently to &quot;yes&quot; on the branch. When we attempted to merge the branch back
to trunk, we were getting merge conflicts, even though we were trying to change
the delivery property value to &quot;yes&quot; on both the trunk and branch. Why was this
a conflict? This didn't seem quite right.</p>
<p>I signed up for the Subversion user's mailing list and made a short post
summarizing our issue. I later learned that it is proper etiquette to attach a
bash script that can demonstrate the problem. Despite this, a Subversion developer
took interest in my post and created a test script in an attempt to reproduce our
issue. At first it looked like he could not reproduce the problem. However another
helpful user pointed out a bug in his script. Once this was fixed, the developer
declared our problem a genuine bug and created a ticket for it in the issue tracker.</p>
</div>
<div class="section" id="the-patch">
<h3>The Patch</h3>
<p>Impressed by all of this, I thanked him for his efforts and tentatively asked if
I could help. The developer told me which file and function he thought the
problem might be in. I downloaded the Subversion source and began looking at the
code. I was fairly impressed with the code quality, so I decided I would try to
create a patch for the bug over the weekend. We really wanted that bug fixed,
and I was genuinely curious to see if I would be able to figure something out.
It would be an interesting challenge and a test of my novice open source skills.</p>
<p>When the weekend came I began a more thorough examination of the Subversion
website. The Subversion team has done a great job in providing documentation on
their development process. This includes a contributing guide and patch
submittal process. I also discovered they had recently added a makefile that
downloaded the Subversion source code and the source for all of Subversion's
dependencies. The makefile then builds everything with debug turned on. Wow! It
took me a few tries to get this working, but the problems were because I did not
have all the development tools installed on my Ubuntu box.  Once this was
sorted, everything went smoothly, and in a matter of minutes I had a Subversion
executable I could run under the gdb debugger. Nice!</p>
<p>I studied the code for about an hour, peeking and poking at a few things in the
debugger. I used the script the developer wrote to recreate the problem. I
wasn't quite sure what I was doing, as I was brand new to this code base. But
the code was clearly written and commented well. My goal was to get a patch that
was in the 80-100% complete range. I wanted to do enough work that a core
developer would be able to see what I was doing and either commit it outright or
easily fill in the parts that I missed. After a while I thought I had a solution
and generated a patch. I sent it to the Subversion developer's mailing list as
per the contributing guide.</p>
</div>
<div class="section" id="the-wait">
<h3>The Wait</h3>
<p>Next I began probably the worst part for a contributor. I had to wait and see if
I got any feedback. On some open source projects a patch may languish for months.
It all depends on the number of developers and how busy they are. My chances
didn't look good as the developers were in the initial stages of getting a
beta version of 1.7 out the door. It was also not clear to me who &quot;owned&quot; the
issue tracker. On some projects, the issue tracker is wide open to the
community. Was I supposed to update the ticket? I wasn't quite sure, and the
contributing guide was silent on this issue. I eventually concluded I was not;
it looked like only committers were using the tracker. Patches were being
discussed on the mailing list instead of in the tracker. This is a bit different
than some projects I am familiar with.</p>
<p>I didn't have to wait long. After a few days, the original developer who
confirmed my bug took interest again. He looked at my patch, and thought I had
missed something. He suggested a change and asked for my opinion. I looked at
the code again; it seemed like a good change and I told him I agreed. I also
warned him I was brand new to the code, and to take my opinion with a grain a
salt. After running my change against the tests, he then committed my patch!
One small victory for open source!</p>
</div>
<div class="section" id="final-thoughts">
<h3>Final Thoughts</h3>
<p>So what went right here? I have to hand it to the Subversion team. They have
been in business a long time, and they have excellent documentation for people
who want to contribute. The makefile they created that sets up a complete
development environment most definitely tipped the scale for me and enabled me
to create my patch. Without that I'm not sure I would have had the time or
patience to get all that unfamiliar source code built. The Subversion team has
really worked hard at lowering the barrier for new contributors.</p>
<p>My advice to people who want to contribute to open source but aren't quite sure
how to go about doing it:</p>
<ul class="simple">
<li>Spend some time reading the documentation. This includes any FAQ's and
contributor guides (if any).</li>
<li>Monitor the user and developer mailing lists to get a feel for how the
community operates. Each project has different customs and traditions.</li>
<li>You may also wish to hang out on the project's IRC channel for the same
reason.</li>
<li>When writing on the mailing lists, be extremely concise and polite.
You don't want to waste anyone's time, and you don't want to
be seen as someone who thinks they are entitled to a fix. Just remember you
are the new guy. You can't just barge in and make demands.</li>
<li>Ask how you can help. Nothing makes a developer happier when someone asks how
they can help. Remember, most of the people in the community are volunteers.</li>
<li>Open source can sometimes be &quot;noisy&quot;. There will be people who
won't quite understand your issue and may hurriedly suggest an incorrect solution or give
incomplete advice. Study their responses and be polite. You may also wish to resist the temptation
to reply right away. This is especially hard when you are new and you don't
know who the &quot;real&quot; developers are. However you should assume everyone is trying to
help.</li>
<li>Finally, be patient. Again, most folks are volunteers. They have real jobs,
families and lives. The project may also be preoccupied with other tasks, like
getting a beta out the door. Now may not be a good time for a brand new
feature, or your bug may not be considered a show stopper to the majority of
the community.</li>
</ul>
<p>A big thank-you to Stefan Sperling from the Subversion team who shepherded my
bug report and patch through their process.</p>
<p>I hope this story encourages you to contribute to open source software!</p>
</div>
</div>
]]></content>
  </entry>
</feed>

