python

I know there are lots of ways to do this. This is not a HOW-TO, but just sharing a script I made for doing this. It's a decent example of how to write a command-line utility in Python.

Since I write my cover letters and resumes in LaTeX I always need to concatenate the two together before sending it to an employer. This is very important so that the person on the other end prints out both and it makes their life easier by not having to worry about two files. For the longest time I just had a simple script that looked like this:

#!/bin/sh

Django Handicap Tracker Screenshot
This is the first Django application I ever built. It was started in the 0.96 days and I have since ported it to 1.0. It mostly works and now supports multiple users using the django-registration and django.contrib.auth module. I haven't done much work on it though. Eventually I will put it online for all to use.

Euler
So far I've solved about 24 of the Project Euler problems. I'm not sure how long it took me, probably a few months working on them here and there. I would like to continue solving more of these when I get the chance. So far I have only implemented them in Python (and in one case, Cython) but I would love to implement them in a language like Haskell one day..or who knows maybe I'll try implementing a few of them in Go

When the Vancouver Sun came out with their Vancouver parking tickets database I immediately had some burning questions, like, did the meter maids work on holidays? Do the work less in the evening than during the day? I found it difficult to answer these questions using their interface, so I decided to screen scrape all 1.6 million parking tickets in to my own MySQL database. This was a bit challenging as they made it difficult to screen scrape the data but eventually it could be done simply by first getting an AppKey, a hidden value inside the HTML source and then doing queries using that AppKey as a parameter. It took about a week to get all 1.6 million tickets downloaded. By using Django, it was easy to get them in to a database and view the results. Initially I just put all the data in to one table, then later I decided to normalize the data a bit which was interesting as I decided to do that in pure SQL which I hadn't done before. I did the scraping itself using a combination of BeautifulSoup, lxml, and mechanize.

vancouver-parking-tickets project at GitHub

MySQL SQL dump (42 MB)

Here's some data:

I was waiting for the January 2010 swimming times to appear on the Vancouver Park Board website but I got tired of all the clicking and scrolling required to see when the lessons were available. The other problem was that once the lessons appeared for one pool, some of the other pools still hadn't posted their lesson schedule for January 2010. The times that came out for the first pool were not ideal so I wanted to wait and see what came out for the other pool, while making sure that the first pool didn't book up.

Django Recipes Screenshot
django-recipes at Google Code

My mom was writing a family cookbook using Microsoft Word and I thought this was a bad idea for several reasons. At first I thought about using LaTeX to separate the style from the content a bit, then I thought about using XML, then I settled on a database as being the most generic to store recipe data. I quickly decided on using Django to create this cookbook framework because Python is probably my strongest language and it makes creating custom websites really easy.

At work I am modifying an existing tool to work from the command line instead of a GUI. Currently everything is a bit coupled to the GUI. On Friday, the next problem I encounted was a global variable in Common.py that was not initialized.

""" Common.py """
def initHSCM():
    global hSCManager
    ...
    hSCManager = win32serviceOpenSCManager(None, None, win32con.SERVICE_ALL_ACCESS)
    ...
 
...
def startService(service):
    """ function that uses hSCManager """
    # These functions don't work when hSCManager is set to None
 
def stopService(service):

I was just looking at some C# code at work today and it had XML Documentation (like javadoc or python docstrings, only with XML). Who was the idiot that came up with that idea? It's the most insane thing I've ever seen. Let's look at the predecessors to C#'s XML documentation:

This script will automatically update a drupal module if your drupal source code is stored in a Subversion respository. It first removes all files except for the .svn directories, then extracts the tarball for the new version of the module. Then it runs an svn status command to see which files are new, which files have been removed in the new version, and which files have changed. The difference between this and svn_load_dirs is that renames aren't handled (well svn_load_dirs doesn't really handle them very well either).

I used this script all the time before Picasa finally added this functionality. This script renames a whole bunch of photos, in a directory for example, appending numbers to the end of a base filename according to the EXIF dates stored inside the JPEGs. For example, a directory full of files that looks like this:

IMG_0123.jpg
IMG_0124.jpg
IMG_0127.jpg
IMG_0128.jpg
...
IMG_0248.jpg
IMG_0250.jpg

could be renamed to this:

Camping Photos_001.jpg
Camping Photos_002.jpg
Camping Photos_003.jpg
Camping Photos_004.jpg
...
Camping Photos_112.jpg

Have you ever experienced a full disk on a server or a desktop? Not fun. This script would normally be run as a cron job and would notify you by email if any drive's free disk space has passed below a certain threshold. The code could be better; I wrote this one a long time ago when I was a bit of a n00b and I was in a rush as well. I might make take a look at it again and see if I can make some improvements.

My uncle came over with a CD of photos from his trip to Peru. I am pretty sure the CD was created with iPhoto as there was an iphoto.xml file in the root of it. Anyways the first problem I had was viewing the pictures after mounting the CD. The file sizes were all weird (too small considering the camera that was used to take them) and some file names were duplicated. Trying to open them up with kuickshow or imagemagick did not work. Then I remembered that this happened before when we got our photos from our wedding photographer (also a Mac user).

Adrian Holovaty gave a talk to the Vancouver Python User's Group tonight. Really nice, funny presentation. His plane was late so while we waited for him we watched his talk at Snakes and Rubies. He showed us some of the new features coming in Django. The neatest thing for me was this new thing he's been working on called "databrowse". It's really awesome. Also the forms stuff is a lot nicer now in the newforms library. Someone asked about migration with Django and how to modify the schema once your app is up and running.

This Sunday, the Vancouver/Zope User's Group (maybe the Zope part should be removed? sorry guys) is having a "Django Jam", a hands-on session where you can see how to create some simple applications or perhaps here some people talk about things they have developed in Django. Unfortunately the two applications I am working on are immature right now and I don't have a laptop, but I'll be there checking out what other people have done.

Tuesday is even more exciting as Adrian Holovaty, the lead Django developer/founder is going to give a talk to the Python User's group while he is in town for another conference. If you are interested in web frameworks (especially simple ones done in a cool language) come and check it out.

Vancouver DjangoJam.

Full invite text follows:

Vancouver's Python and Zope User Group will be having a talk on Python web frameworks, ie. Django and Turbogears at their upcoming meeting on October 3rd. I'm looking forward to learning about web frameworks in general a bit more and perhaps what differentiates them from each other and from Ruby on Rails.