Learn one new language a year

Sun, 07 Jun 2009 00:00:00 GMT

June 7 2009 - East Oakland, Ca

Taking the advice from the Pragmatic Programmer I set out to learn Haskell this year. My plan didn't turn out like I expected. I started learning Haskell by working through Real World Haskell. It was tough going, so I decided I wanted to start blogging about it. It was than that I remembered that my blog has been neglegled for a long time (more than a year and maybe 2) and that I had been on the lookout for a different blogging platform for some time.

I had recently found jekyll (a static site generator written in Ruby) from a reference on HN and thought it was a great idea. This is exactly the kind of thing I was looking for. But using jekyll didn't feel quite (well, sortof-- the layout and some of the styling does come from jekyll) right since I had often told myself that I wanted to write my own blogging tool. So I dropped learning Haskell (for the moment) and set out to write my own blogging tool.

My background is mostly in Perl, PHP, and little bit of Ruby. But over the last year I've been creating a few tools in Python for automating some day-to-day operations at work. I chose Python at work after reading Why Python? and having similar experiences. The small scripts I would write often worked right out of the editor. I also appreciated how readable the code was. So I started hacking away at a blogging tool using Python.

I'm calling the tool furthermore. You can download or take a look at github. Furthermore is pretty simple. It is a bare bones static site generator. Here is a brief summary of the features:

mako templates for python-based template generation
pygments syntax highlighting (via markdown)
markdown for text-to-html generation
a local http server for site previewing

To get started, you will need the following:

Pygments for syntax highlighting. Also see the codehighlight markdown plugin.
PyRSS2Gen for RSS feed generation
PyYAML for documents

Posts are contained in the "posts" directory. Each post consists of a YAML preamble that includes the template type and the post title. Post files themselves follow the format YYYYMMDD-post-slug-goes-here.markdown. The year, month, day, and slug will be parsed out of the file name to produce a url of the form /archives/YYYY/MM/DD/slug.html. The preamble looks something like this.

---
title: Learn one new language a year
layout: post
...

Everything after the close of the YAML preamble (...) is treated as the document contents in markdown. The layout value will be used to lookup the proper template in the "templates" directory and the title value will be plugged into the template to generate the title of the page. At the top level you will also find a "images" and "css" directories containing the expected collateral.

And finally, to render your posts and any other files, run furthermore:

1	./bin/furthermore -P -b . -o www -f index.html -S

This will process all of your posts using the current directory as the base (where to find css, images, posts, and templates directories), while outputting the rendered content to www. It will also run index.html through mako and startup a local webserver so you can preview the site. To see all of the options provided, run furthermore with -h.

If you get stuck take a look at the usage document I'm putting together. I'm still updating, but it should have enough to get the idea and get started if you want to use it.

Enjoy!

Exploring the relative performance of different list operations in Python

Wed, 17 Jun 2009 00:00:00 GMT

June 17 2009 - East Oakland, Ca

This past weekend, I attended the Python training session at USENIX09 in San Diego taught by David Beazley. He posted his slides and examples on his web site.

One question that came to mind during the session was which form of list processing is faster: a for loop or a list comprehension. The general consensus in the class (and from David) was that list comprehensions should be faster. The normal caveats of context apply.

If you aren't familiar with list comprehensions, they take the following form in python:

1 2	>>> [val for val in xrange(0,10)] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

David mentioned that list comprehensions in Python were inspired by Haskell. The Haskell form of the above looks like this:

1 2	Prelude> [x \| x <- [0..9]] [0,1,2,3,4,5,6,7,8,9]

To explore this further, I came up with a simple scenario that you might encounter in working with lists. I pulled down part (about 10M) of an apache access log and came up with some small segment of each log that I wanted to pull out. For this experiment, I just pull out the first four elements of each log entry. A sample might look like this:

192.168.0.9 - - [04/Sep/2008:00:21:18 +0000] "GET / HTTP/1.1" 200 2029 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1"

And the parsed out pieces look like this:

['192.168.0.9', '-', '-', '[04/Sep/2008:00:21:18 +0000]']

I pull out these elements by first slicing the first 45 characters from the line and them splitting by the first 3 spaces (note that this might truncate part of the timestamp field). Here is the snippit:

1	line[:45].split(' ', 3)

Before I moved ahead, I googled a little about the relative performance of list comprehensions and for loops in Python. I found an article by Guido called Python Patterns - An Optimization Anecdote where he discusses some of the challenges in trying to optimize converting a list of integers into a string. Although I'm not trying to optimize anything in my experiment, his essay provided some insights into how one might approach optimizing python code. He also provides the code he used for his different iterations and the function he used to do the timing of each version. I used his timing function in running my tests.

I started by creating a function to loop through the lines in the file using a for loop. Here is the function:

def forloop_listnf(lines):
    records = []
    for line in lines:
        records.append(line[:45].split(' ', 3))
    return records

It just loops through each line, pulls out the first three elements and appends it to a list. The next function uses a list comprehension to do the same thing:

1 2	def lc_listnf(lines): return [line[:45].split(' ', 3) for line in lines]

And finally, another approach to processing lists is using the map() function. Since this is also a common approach (and should be done in C per Guido's article) I included a function to do the same using map(). One difference with the map() implementation is that a separate function was need to do the splice and split operation on each element in the array provided to map().

def do_split(line):
    return line[:45].split(' ', 3)

def map_list(lines):
    return map(do_split, lines)

The result surprised me a little. The map() implementation was the slowest. The for loop was about 5% faster and the list comprehension was about 10% faster than map().

I went through a couple of iterations before I settled on the functions above. I first used the do_split for all three but I later decided that this unnecessarily slows down (with an extra lookup and function call) the for loop and the list comprehension because they can do the splice and split in-line. I tried to see if a lambda would improve the map() performance, but it performed about the same. For this simple test map() may have a handicap due to the function call on each element.

Conclusion

As a noob to python I'm not in a position to say conclusively that list comprehensions are faster than for loops or the map() function. This experiment seems to suggest that for simple cases such as this, the list comprehension is faster. It also looks cleaner (IMO). Building the list can be done clearly and concisely on one line. Note also that list comprehensions can be nested. An initial path that I started on with this experiment was adding one more pass of processing over the split out fields. A nested list comprehension worked, but I backed off of that path to keep it simple.

Why are they faster? My initial thought is that there is less interpreter overhead with the list comprehension. In the for loop, you need one other variable (the result array) and you need to lookup this variable during each iteration of the loop. You may not have that overhead with the list comprehension (or perhaps it is reduced). You can find the code to these samples here.

David's training classes are good. You can find out more here.

damonsnyder.com RSS Feed

Learn one new language a year

Exploring the relative performance of different list operations in Python

Conclusion