msgbartop
by Brian Neal
msgbarbottom

31 Dec 09 Python and SSH

I was fooling around with my webcam, and I got to wondering if I could write a small Python script to upload a snapshot to my webserver periodically. Since I was planning on uploading a new snapshot every ten minutes or so, I’d rather use SFTP to avoid sending my login and password unencrypted so frequently. I had already used FTP in Python before in a backup script, but I didn’t know if the Python standard library had a solution for SSH. After a bit of Googling, I discovered the 3rd party Python module Paramiko (the name is a combination of the Esperanto words for “paranoid” and “friend” — I love that), which allows you to do all kinds of SSH2 operations, including SFTP. Paramiko is extremely well documented (thank-you!) and I had no problems at all using it to accomplish my task. I also found this great introductory article by Python core developer Jesse Noller that was helpful in getting up to speed on it quickly.

Since I was using my webcam from Windows XP, I decided to write my script under the new 1.7.1 version of Cygwin that came out recently. You’ll need to get both the Paramiko package and the python-crypto package.

Happy New Year everyone!

Tags: , , , , ,

20 Dec 09 SG101 2.0 Status Report

This is the obligatory “why haven’t you been blogging about your project in a while” post. Yes, I suppose it is time to give a quick update.

Things slowed down dramatically on SG101 2.0 this summer. There was the 2009 SG101 convention vacation and other fun summer things to distract me. I had put up a beta site for feedback but was still missing a forums application. I got the itch to start back up again sometime in the Fall. I decided to start on a forums application myself and to see how it went. If it was too complicated I would look for a third-party solution.

This is one case where I did look at several third-party applications for ideas and to check on their status. There doesn’t seem to be a single recognized forums application in Django-land. There are a number of them, and they range from very simple to moderately complex. Many of them seem unsupported and have obvious problems. So in the end, I decided that since the forums are probably the most important part of my site, it would be best if I wrote it myself so that I could understand it completely. This includes both strengths and weaknesses. I did borrow many ideas from existing applications, and some of my initial momentum came from djangobb. However, I quickly stopped looking at other apps because Django really makes it easy to write complex web applications once you get an idea and try a few things.

My forum app contains most of the functionality of the venerable phpBB-based board I have now. I added a few things like the ability for users to flag posts as spam or abuse (I sure wish I had that now). I am considering making the first few posts of a user require approval to counter spam. But I’m not sure it is worth the effort with the “flag post” feature in-place. I might just wait and see how well that works.

I also decided to save a user’s post read and unread status in the database, instead of using cookies. Too many of my existing users complain that when their cookies expire they lose track of which threads are new. It will cost some database space, no doubt, but it is an often requested feature to fix this issue. I implemented a rolling 7-day window of thread and post read status, and in initial tests it seems to work just fine. It did add significant complexity to the design however, and I’m not looking forward to debugging that logic when a problem occurs.

After finishing the forums, I began working on my lengthy to-do list using my Trac issue tracker. I also spent a great deal of time refactoring some of my original code that I wrote over a year ago. I’ve become so much more proficient with Python and jQuery it is inevitable. My task list has become quite small and I am thinking about wiping the existing beta site and putting up a new one over holiday break and launching an official beta test.

The one area that I am lacking in right now is a good design and layout. A few users have volunteered to help with that, and one in particular is showing me some really nice work. If I can just manage to implement his design we may be on to something. I may also try to reach out to someone who is familiar with Django.

There are a couple of interesting problems I either solved or worked around during this period that I should blog about. I’ll just have to find the time to do that. In particular, I wanted to share how I created an admin dashboard for user-created content that needs admin approval before being published.

I’ve also volunteered to give a “brown-bag” lunchtime talk at my employer on Python. I’ll have to prepare some slides over the holiday break for this.

Tags: ,

13 Dec 09 Python Subprocess Popen and Windows

I was writing a Python script to automate some Subversion-related tasks at work last week on Windows. The Python code was calling the Subversion command-line client applications using the Python subprocess module. I don’t know why exactly, but I had passed in the parameter shell=True to the subprocess.Popen function. I probably did this because I saw a lot of examples on the web, and you know, monkey-see, monkey-do. In hindsight, these examples were probably for Unix. It turned out that this caused a lot of problems. The Subversion clients started complaining when handed filenames that contained “special” characters like ampersands. And when they got to the files that had both ampersands and spaces, all kinds of crazy errors were produced. I was pulling my hair out, trying to escape the special characters and/or use quotes around the filename, but I could not come up with a combination that would work for all the different filenames the script encountered. Finally, as I stared at the code, I somehow got the idea to try switching that shell=True to shell=False.  Suddenly, everything worked perfectly. I undid all the escaping and quoting and everything worked as expected.

So I’m not sure what shell=True does on Windows, and it certainly isn’t documented. I do not recommend using it unless you know what is happening  in the implementation. Interestingly, shell=False is the default. I should have left it that way. :)

Tags: , , ,