Installing omniORBpy in a virtualenv
I came across an interesting question on Stackoverflow, which was essentially How to install omniORBpy in a virtualenv? I've had a lot of experience with omniORB, a high quality, open-source CORBA ORB, at work. omniORB is a C++ ORB, but it also includes omniORBpy, a thin Python wrapper around the C++ core. Despite using omniORBpy extensively, I had never used it in a virtualenv.
So I got curious, and gave it a shot. You can read my answer, which the questioner accepted. I'd love to hear of any better ways of doing it. I worked it out for Ubuntu, but a Windows solution would be nice to see also.
Upgrading Trac on Windows Gotcha - Part 2
I have previously reported on some problems I had when upgrading our Trac install at work. Today I attempted another upgrade to Subversion 1.7.4 and Trac 0.12.3 on Windows. I upgraded to Python 2.7.2 along the way. I ran into another problem with the Python bindings to Subversion which took a while to figure out. I had no problems upgrading Subversion, but Trac could not see our repository. The symptoms were that the Trac "timeline" and "browse source" features were missing.
Following the Trac troubleshooting advice, I opened an interactive Python session and tried this:
E:\Trac_Data>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> from svn import client
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\svn\client.py", line 26, in <module>
from libsvn.client import *
File "C:\Python27\lib\site-packages\libsvn\client.py", line 25, in <module>
_client = swig_import_helper()
File "C:\Python27\lib\site-packages\libsvn\client.py", line 21, in swig_import
_helper
_mod = imp.load_module('_client', fp, pathname, description)
ImportError: DLL load failed: The operating system cannot run %1
After some head scratching and googling I finally found the problem. I had used the Windows .msi installer, graciously provided by Alagazam, aka David Darj, to install Subversion. This placed the Subversion binaries and DLL's in C:\Program Files\Subversion\bin. I then unzipped the Python 2.7 bindings to the C:\Python27\Lib\site-packages folder. The bindings depend on the DLL's in the Subversion\bin folder. But unfortunately for me, there were already two older versions of the DLL's, libeay32.dll and ssleay32.dll on my path. So when the bindings went looking for those two DLL's, instead of finding them in Subversion\bin, it found the older versions somewhere else.
To fix this, you can either rearrange your path, or copy those two DLL's to your Python27\Lib\site-packages\libsvn folder. In the future, I am going to just copy all the DLL's from Subversion\bin to the libsvn folder.
I examined the pre-built Subversion packages from Bitnami and CollabNet. They had packaged all of the Subversion DLL's with the Python bindings together in the same directory, so this seems reasonable. Later, on the Subversion users' mailing list, Alagazam gave the nod to this approach.
A big thank you to Alagazam for the help and for the Windows binaries. And of course thanks to the Apache, Subversion, Trac, & Python teams for making great tools.
A TeamSpeak 3 viewer with Python & Javascript
The Problem
My gaming clan started using TeamSpeak 3 (TS3) for voice communications, so it wasn't long before we wanted to see who was on the TS3 server from the clan's server status page. Long ago, before I met Python, I had built the clan a server status page in PHP. This consisted of cobbling together various home-made and 3rd party PHP scripts for querying game servers (Call of Duty, Battlefield) and voice servers (TeamSpeak 2 and Mumble). But TeamSpeak 3 was a new one for us, and I didn't have anything to query that. My interests in PHP are long behind me, but we needed to add a TS3 viewer to the PHP page. The gaming clan's web hosting is pretty vanilla; in other words PHP is the first class citizen. If I really wanted to host a Python app I probably could have resorted to Fast CGI or something. But I had no experience in that and no desire to go that way.
I briefly thought about finding a 3rd party PHP library to query a TS3 server. The libraries are out there, but they are as you might expect: overly complicated and/or pretty amateurish (no public source code repository). I even considered writing my own PHP code to do the job, so I started looking for any documentation on the TS3 server query protocol. Luckily, there is a TS3 query protocol document, and it is fairly decent.
But, I just could not bring myself to write PHP again. On top of this, the gaming clan's shared hosting blocks non-standard ports. If I did have a PHP solution, the outgoing query to the TS3 server would have been blocked by the host's firewall. It is a hassle to contact their technical support and try to find a person who knows what a port is and get it unblocked (we've had to do this over and over as each game comes out). Thus it ultimately boiled down to me wanting to do this in Python. For me, life is too short to write PHP scripts.
I started thinking about writing a query application in Python using my dedicated server that I use to host a few Django powered websites. At first I thought I'd generate the server status HTML on my server and display it in an <iframe> on the gaming clan's server. But then it hit me that all I really needed to do is have my Django application output a representation of the TS3 server status in JSON, and then perhaps I could find a slick jQuery tree menu to display the status graphically. I really liked this idea, so here is a post about the twists and turns I took implementing it.
The Javascript
My searching turned up several jQuery tree menu plugins, but in the end I settled on dynatree. Dynatree had clear documentation I could understand, it seems to be actively maintained, and it can generate a menu from JSON. After one evening of reading the docs, I built a static test HTML page that could display a tree menu built from JSON. Here the Javascript code I put in the test page's <head> section:
var ts3_data = [
{title: "Phantom Aces", isFolder: true, expand: true,
children: [
{title: "MW3", isFolder: true, expand: true,
children: [
{title: "Hogan", icon: "client.png"},
{title: "Fritz!!", icon: "client.png"}
]
},
{title: "COD4", isFolder: true, expand: true,
children: [
{title: "Klink", icon: "client.png"}
]
},
{title: "Away", isFolder: true, children: [], expand: true}
]
}
];
$(function(){
$("#ts3-tree").dynatree({
persist: false,
children: ts3_data
});
});
Note that client.png is a small icon I found that I use in place of dynatree's default file icon to represent TS3 clients. If I omitted the icon attribute, the TS3 client would have appeared as a small file icon. Channels appear as folder icons, and this didn't seem to unreasonable to me. In other words I had no idea what a channel icon would look like. A folder was fine.
With dynatree, you don't need a lot of HTML markup, it does all the heavy lifting. You simply have to give it an empty <div> tag it can render into.
<body>
<div id="ts3-tree"></div>
</body>
</html>
Here is a screenshot of the static test page in action.
Nice! Thanks dynatree! Now all I need to do is figure out how to dynamically generate the JSON data and get it into the gaming clan's server status page.
The Python
Looking through the TS3 protocol documentation I was somewhat surprised to see that TS3 used the Telnet protocol for queries. So from my trusty shell I telnet'ed into the TS3 server and played with the available commands. I made notes on what commands I needed to issue to build my status display.
My experiments worked, and I could see a path forward, but there were still some kinks to be worked out with the TS3 protocol. The data it sent back was escaped in a strange way for one thing. I would have to post-process the data in Python before I could use it. I didn't want to reinvent the wheel, so I did a quick search for Python libraries for working with TS3. I found a few, but quickly settled on Andrew William's python-ts3 library. It was small, easy to understand, had tests, and a GitHub page. Perfect.
One of the great things about Python, of course, is the interactive shell. Armed with the TS3 protocol documentation, python-ts3, and the Python shell, I was able to interactively connect to the TS3 server and poke around again. This time I was sitting above telnet using python-ts3 and I confirmed it would do the job for me.
Another evening was spent coding up a Django view to query the TS3 server using python-ts3 and to output the channel status as JSON.
from django.conf import settings
from django.core.cache import cache
from django.http import HttpResponse, HttpResponseServerError
from django.utils import simplejson
import ts3
CACHE_KEY = 'ts3-json'
CACHE_TIMEOUT = 2 * 60
def ts3_query(request):
"""
Query the TeamSpeak3 server for status, and output a JSON
representation.
The JSON we return is targeted towards the jQuery plugin Dynatree
http://code.google.com/p/dynatree/
"""
# Do we have the result cached?
result = cache.get(CACHE_KEY)
if result:
return HttpResponse(result, content_type='application/json')
# Cache miss, go query the remote server
try:
svr = ts3.TS3Server(settings.TS3_IP, settings.TS3_PORT,
settings.TS3_VID)
except ts3.ConnectionError:
return HttpResponseServerError()
response = svr.send_command('serverinfo')
if response.response['msg'] != 'ok':
return HttpResponseServerError()
svr_info = response.data[0]
response = svr.send_command('channellist')
if response.response['msg'] != 'ok':
return HttpResponseServerError()
channel_list = response.data
response = svr.send_command('clientlist')
if response.response['msg'] != 'ok':
return HttpResponseServerError()
client_list = response.data
# Start building the channel / client tree.
# We save tree nodes in a dictionary, keyed by their id so we can find
# them later in order to support arbitrary channel hierarchies.
channels = {}
# Build the root, or channel 0
channels[0] = {
'title': svr_info['virtualserver_name'],
'isFolder': True,
'expand': True,
'children': []
}
# Add the channels to our tree
for channel in channel_list:
node = {
'title': channel['channel_name'],
'isFolder': True,
'expand': True,
'children': []
}
parent = channels[int(channel['pid'])]
parent['children'].append(node)
channels[int(channel['cid'])] = node
# Add the clients to the tree
for client in client_list:
if client['client_type'] == '0':
node = {
'title': client['client_nickname'],
'icon': 'client.png'
}
channel = channels[int(client['cid'])]
channel['children'].append(node)
tree = [channels[0]]
# convert to JSON
json = simplejson.dumps(tree)
cache.set(CACHE_KEY, json, CACHE_TIMEOUT)
return HttpResponse(json, content_type='application/json')
I have to make three queries to the TS3 server to get all the information I need. The serverinfo command is issued to retrieve the TS3 virtual server's name. The channellist command retrieves the list of channels. The clientlist command gets the list of TS3 clients that are currently connected. For more information on these three commands see the TS3 query protocol document.
The only real tricky part of this code was figuring out how to represent an arbitrary, deeply-nested channel tree in Python. I ended up guessing that cid meant channel ID and pid meant parent ID in the TS3 query data. I squirrel away the channels in a channels dictionary, keyed by channel ID. The root channel has an ID of 0. While iterating over the channel list, I can retrieve the parent channel from the channels dictionary by ID and append the new channel to the parent's children list. Clients are handled the same way, but have different attributes. By inspecting the clientlist data in the Python shell, I noticed that my Telnet client also showed up in that list. However it had a client_type of 1, whereas the normal PC clients had a client_type of 0.
I decided to cache the results for 2 minutes to reduce hits on the TS3 server, as it has flood protection. This probably isn't needed given the size of our gaming clan, but Django makes it easy to do, so why not?
Putting it all together
At this point I knew how to use my Django application to query the TS3 server and build status in JSON format. I also knew what the Javascript and HTML on the gaming clan's server status page (written in PHP) had to look like to render that JSON status.
The problem was the server status page was on one server, and my Django application was on another. At first I thought it would be no problem for the Javascript to do a GET on my Django server and retrieve the JSON. However I had some vague memory of the browser security model, and after some googling I was reminded of the same origin policy. Rats. That wasn't going to work.
I briefly researched JSONP, which is the technique that Facebook & Google use to embed those little "like" and "+1" buttons on your web pages. But in the end it was just as easy to have the PHP script make the GET request to my Django application using a file_get_contents() call. The PHP can then embed the JSON directly into the server status page:
$ts3_source = 'http://example.com/ts3/';
$ts3_json = file_get_contents($ts3_source);
require_once 'header.php';
And in header.php, some HTML sprinkled with some PHP:
<script type="text/javascript">
var ts3_data = <?php echo $ts3_json; ?>;
$(function(){
$("#ts3-tree").dynatree({
persist: false,
children: ts3_data
});
});
</script>
That did the trick. In the end I had to touch a little PHP, but it was tolerable. That was a very round-about solution to building a TS3 viewer in Python and Javascript. While I doubt you will have the same strange requirements that I had (multiple servers), I hope you can see how to combine a few technologies to make a TS3 viewer in Python.
Who's Online with Redis & Python, a slight return
In a previous post, I blogged about building a "Who's Online" feature using Redis and Python with redis-py. I've been integrating Celery into my website, and I stumbled across this old code. Since I made that post, I discovered yet another cool feature in Redis: sorted sets. So here is an even better way of implementing this feature using Redis sorted sets.
A sorted set in Redis is like a regular set, but each member has a numeric score. When you add a member to a sorted set, you also specify the score for that member. You can then retrieve set members if their score falls into a certain range. You can also easily remove members outside a given score range.
For a "Who's Online" feature, we need a sorted set to represent the set of all users online. Whenever we see a user, we insert that user into the set along with the current time as their score. This is accomplished with the Redis zadd command. If the user is already in the set, zadd simply updates their score with the current time.
To obtain the curret list of who's online, we use the zrangebyscore command to retrieve the list of users who's score (time) lies between, say, 15 minutes ago, until now.
Periodically, we need to remove stale members from the set. This can be accomplished by using the zremrangebyscore command. This command will remove all members that have a score between minimum and maximum values. In this case, we can use the beginning of time for the minimum, and 15 minutes ago for the maximum.
That's really it in a nutshell. This is much simpler than my previous solution which used two sets.
So let's look at some code. The first problem we need to solve is how to convert a Python datetime object into a score. This can be accomplished by converting the datetime into a POSIX timestamp integer, which is the number of seconds from the UNIX epoch of January 1, 1970.
import datetime
import time
def to_timestamp(dt):
"""
Turn the supplied datetime object into a UNIX timestamp integer.
"""
return int(time.mktime(dt.timetuple()))
With that handy function, here are some examples of the operations described above.
import redis
# Redis set keys:
USER_SET_KEY = "whos_online:users"
# the period over which we collect who's online stats:
MAX_AGE = datetime.timedelta(minutes=15)
# obtain a connection to redis:
conn = redis.StrictRedis()
# add/update a user to the who's online set:
username = "sally"
ts = to_timestamp(datetime.datetime.now())
conn.zadd(USER_SET_KEY, ts, username)
# retrieve the list of users who have been active in the last MAX_AGE minutes
now = datetime.datetime.now()
min = to_timestamp(now - MAX_AGE)
max = to_timestamp(now)
whos_online = conn.zrangebyscore(USER_SET_KEY, min, max)
# e.g. whos_online = ['sally', 'harry', 'joe']
# periodically remove stale members
cutoff = to_timestamp(datetime.datetime.now() - MAX_AGE)
conn.zremrangebyscore(USER_SET_KEY, 0, cutoff)
Upgrading Trac on Windows Gotchas
At work, we are outfitted with Windows servers. Despite this obstacle, I managed to install Trac and Subversion a few years ago. During a break in the action, we decided to update Subversion (SVN) and Trac. Since we are on Windows, this means we have to rely on the kindness of strangers for Subversion binaries. I ran into a couple of gotchas I'd like to document here to help anyone else who runs into these.
I managed to get Subversion and Trac up and running without any real problems. However when Trac needed to access SVN to display changesets or a timeline, for example, I got an error:
tracerror: unsupported version control system "svn" no module named _fs
After some googling, I finally found that this issue is documented on the Trac wiki, but it was kind of hard to find. To fix this problem, you have to rename the Python SVN binding's DLLs to *.pyd. Specifically, change the libsvn/*.dll files to libsvn/*.pyd, but don't change the name of libsvn_swig_py-1.dll. I'd really like to hear an explanation of why one needs to do this. Why doesn't the Python-Windows build process do this for you?
The second problem I ran into dealt with mod_wsgi on Windows. Originally, a few years ago, I setup Trac to run under mod_python. mod_python has long been out of favor, so I decided to cut over to mod_wsgi. On my Linux boxes, I always run mod_wsgi in daemon mode. When I tried to configure this on Windows, Apache complained about an unknown directive WSGIDaemonProcess. It turns out that this mode is not supported on Windows. You'll have to use the embedded mode on Windows.
Next Page »
