Code Kills

Vim functions to change tab treatment

Sun, 22 Jan 2012 16:00:00 EST

I've gotten tired of doing the :setlocal tabstop=... dance every time I start editing a file from someone who has their own unique feelings about how wide tabs should be... So I've written two little functions which make it easy to change a buffer's tab mode:

For example, when I start editing something written by someone who believes spaces are evil and tabs should be five spaces wide, I just need to :call HardTabs(5).

In case of Wikimergency (removing Wikipedia's blackout)

Wed, 18 Jan 2012 02:06:12 EST

Here's a bookmarklet that will remove the SOPA blackout overlay from Wikipedia:


javascript:(function(){document.getElementById("mw-sopaOverlay").style.display = "none"; var ids = ["mw-page-base", "mw-head-base", "content", "mw-head", "mw-panel", "footer"]; for (var i = 0; i < ids.length; i += 1) document.getElementById(ids[i]).style.display = "block";})()

Or bookmark this link.

Loading Google Closure libraries from Node.js

Tue, 10 Jan 2012 01:24:42 EST

While developing remora (which will be released "soon") I had to load Google Closure libraries from Node.js. As it turns out, I'm the first one who has had to do this, so a blog post seemed in order.

I've loaded my Closure library in Node by creating a Node module which loads Closure's base.js, tells Closure how to load JavaScript files, loads my Closure library, then exports everything (the goog name space and my namespace) through module.exports.

First, an execfile function is needed, which will load and evaluate a JavaScript file using a particular namespace:

var vm = require("vm");
var fs = require("fs");

function execfile(path) {
  var data = fs.readFileSync(path);
  vm.runInThisContext(data, path);
}

Note: for various reasons (one of which being that you can't control the value of 'this' from the 'vm' module) vm.runInThisContext must be used to load the scripts into the global context. Trying to load the into a sandbox seems to fail for reasons I can't fully explain yet.

Next, the Closure base.js and deps.js files are loaded and CLOSURE_IMPORT_SCRIPT is set to a thin wrapper around execfile:

var base_basedir = "../src/browser/";
execfile(base_basedir + "base.js");
execfile(base_basedir + "deps.js");
goog.global.CLOSURE_IMPORT_SCRIPT = function(path) {
  execfile(base_basedir + path);
  return true;
};

Now Closure libraries which use goog.require(...) and goog.provide(...) can be loaded:

execfile("../src/remora.js");

(note: only the base library needs to be loaded; its dependencies will be taken care of by goog.require(...)).

Normal JavaScript packages can also be loaded:

execfile("../libs/underscore-1.2.3.js");

Finally, module.exports is set:

module.exports = {
    goog: GLOBAL.goog,
    remora: GLOBAL.remora,
};

Note: because Closure scripts expect that require/provide will inject items into the global namespace, the global names created (ex, goog and remora) cannot be deleted from the global namespace.

And that's all there is to it. This file can now be loaded like any other Node module (ex, var my_closure_library = require("./closure_compat.js")).

For a complete example, see remora/src/node/remora.dev.js.

Which SSD You Should Buy

Tue, 22 Nov 2011 19:32:25 EST

I'm thinking about buying a 200+ GB SSD for my laptop, and this post summarizes my research. Note that it is current as of November 22, 2011 and will likely be irrelevant by 2012.

tl;dr The Corsair Force GT seems to be the fastest choice with Crucial m4 coming in a fairly close second. Data on failure rates for these drives/controllers has been hard to find, though, and all signs point to Intel drives being the most reliable (but also most expensive and slowest).

TechReport's SSD performance review which @samstokes sent me, which seems to find the two aforementioned drives to have the best performance, with the Corsair Force Series 3 making a decent showing as well. It also suggests that the Intel 500 series is a bit slower and significantly more expensive than the competition.

Tom's Hardware investigation on SSD reliability suggests that Intel's SSDs are much more reliable (specifically their X25-M series; a sentiment I've read in other places too), but it doesn't compare their failure rates to other manufacturers because "those are the drives that big businesses currently trust the most".

Friends on Twitter and Facebook generally recommended the m4.

It's also worth noting that I've read quite a few posts similar to The Hot/Crazy Solid State Drive Scale which suggest that failures can be fairly common. These posts have mostly been more than six months old, though, so I'm hoping that most of these problems have been sorted out in newer drives (it seems that there are often issues which can be fixed by a software update).

The net result is that I'm probably going to get a Force GT or m4, which ever I can find for a reasonable price from a local retailer with a good return policy.

Disclaimer: what you see is what you get. I've done enough research to convince myself that an SSD isn't a terrible idea, but that's definitely far from enough research to objectively show that an SSD isn't a terrible idea. Also, I've included links to Amazon for two reasons: 1) because prices change so often that quoting them here wouldn't be that useful, and 2) I've just setup a fancy affiliate account, so I stand a small chance of making a small amount of money from them.

howto: running the wifi and power at medium sized tech events

Mon, 07 Nov 2011 13:32:36 EST

Because I've done it a few times now and picked up a few tricks, I thought I would write a checklist for anyone who needs (or wants) to run the wifi and power at a medium sized (10 - 100 people) tech event.

I hope that this checklist will be a useful for events like Software Carpentry or Ladies Learning Code, as it will give someone with some networking knowledge [0] the nuts-and-bolts they need to setup and run a great (or at least functional) network.

I've published this document as a Gist on GitHub to make updates easy.

The document: Wifi and power for medium sized tech events: https://gist.github.com/1346886

[0]	anyone who knows the function of DHCP and the meaning of "default gateway" should be fine.

Winter protip: hand warmers. Lots of them.

Mon, 31 Oct 2011 00:08:47 EDT

Now that winter is coming, I'd like to share a tip that has been the difference between a terrible and a great day: hand warmers. Lots of hand warmers.

Here's what you do:

Buy hand warmers. Lots of them. Like fifteen pairs of them. They are $1.20 for a pair, so no banks will be broken by having a few extras.
Put a pair (or two?) in the pocket of each of your winter jackets.
Use them! Next time you (or your friend/spouse/child) is uncomfortably cold, just rip a hand warmer open. Ten minutes later you'll have a warm little bundle of joy to stick in your boot/glove/jacket/hat.

There are a bunch of reasons I've found step three to be hard: I don't want to admit to myself that I'm actually cold, I don't want to use them now because I may need them even more in the future, or because I don't feel like the warmth is worth the $1.20 it will cost... But since I have never regretted using one, I feel like it's worth tricking myself into using them more liberally by keeping lots of spares around, using it "for a friend", and telling myself that each one only costs 60¢ (that's like two quarters!).

On Startups and Identity

Tue, 18 Oct 2011 15:18:42 EDT

A curious thing I've noticed recently is that a large number of startups don't include any information about the identity of the founders/employees.

I find this strange because, in my opinion, a big advantage of working with (or using the product of) a startup is that I can get to know the person (or people) behind the scenes. They might even be someone I know, or someone a friend knows.

I would go on, but I would just be repeating what Jason Cohen says in You're a little company, now act like one.

So, if you're part of a startup: please, include your identity and the identity of the other founders/employees somewhere on your website.

What Makes a Good Resume

Mon, 17 Oct 2011 21:40:54 EDT

I've read a small but nontrival number of resumes in my life, and each time I've vocalized strong feelings about what makes a resume good or bad.

Some people, presumably mistaking these strong feelings for expertise, have asked me for advice when writing their resume... So I will summarize here the things which I appreciate or despise while reading a resume.

Disclaimer: these are just my feelings. It's likely that people similar to me will have similar feelings, but it's unlikely that someone like me will be the first one reading your resume.

For many positions a non-technical recruiter or HR person will be the first to see your resume. They will likely apply naive pattern matching against a given set of buzzwords to determine the quality of applicants, so a section containing buzzwords and synonyms (ex, "Python, CPython, Django 1.3, Django 1.2, Django forms, Django admin, AJAX, ...") would likely be helpful (although, again, I can't speak with authority as I'm not a non-technical recruiter).

Steve Hanov has a comic that might be helpful.

As I'm looking at a resume I'm trying to answer three questions: what does the applicant know? How well do they know it? Would an interview with this applicant be a waste of time?

Some of the things I like to see are:

A summary of skills, programming languages and tools sorted by experience. I can't count the number of times a applicant includes a list like this on their resume:

Programming languages:

Python, Java, C/C++, Objective-C, JavaScript, C#, Visual Basic, HTML, CSS, Perl, Scheme, ML, Prolog.

Tools:

Make, Vim, Emacs, Word, Power Point, Terminal, Linux, Firefox, Flash Player.

This kind of list is absolutely useless. Actually, no, that's not true. It's very useful. It tells me that the applicant doesn't feel the need to distinguish between their knowledge of Prolog and their knowledge of JavaScript, and they think C is similar enough to C++ to warrant listing them together.

A list ordered by experience is much more useful. For example, my list might look something like this:

Programming languages:

Python (5 years, 50k SLOC), ActionScript (2 years, 20k SLOC), JavaScript (5 years, 5k SLOC), HTML+CSS (5 years), C (2 years, 2k SLOC), some knowledge of PHP, Scheme, Haskell, and ML.

Tools:

Vim (6 years), Bash (4 years), POSIX shell environment and utilities (4 years), Linux (8 years, mostly Debian and server administration), Mercurial (3 years), git (1 year).

Alternatively, the headings "most experience with" and "some experience with" could be used, as they give some idea of both expertise and breadth.

It's also worth noting that, at least in students, self assessment of knowledge is useless, so when I see "export" or "very good" in a resume I mentally replace it with "dangerously ignorant".

Relevant links. I feel like this should go without saying, but I would guess that less than 10% of the resumes I have read included any kind of link.

I appreciate any links to a applicant's online presence: a blog or personal website (if it has been updated in the last five years) is good, and an account on GitHub, BitBucket, StackOverflow, Reddit or other social site would be even better. It's hard to convey interpersonal and communication skills on a resume, but social sites serve as a good way to demonstrate those skills.

And a corollary which, based on experience, isn't as obvious as you might expect: irrelevant links (for example, to a "home page" without any interesting content) are worse than useless - they waste my time and make the applicant look silly.

Length. I don't care. After the summary of skills, languages and tools I'm usually skimming for interesting things... And if the applicant has three pages worth of interesting things, I would like to read those three pages. But if the applicant has three pages of boring things, I will probably skim the second and glance over the third.

Relevant work experience. And, more specifically, specific work experience. A statement like "worked on a team" or "helped implement" isn't as helpful as "wrote server-side JavaScript" or "designed the database schema". Also, while others might, I don't care about unrelated work experience.

It's also useful to include the technologies used at a job. For example: "MegaCorp - 2001 - 2003; - Designed database schema to optimize sale transactions; - Performed A/B testing to optimize widget sales; Technologies: PostgreSQL, PHP, libABTestOMatic"

School. As a university dropout, I'm a bit biased here, but I care very little about school or degrees unless they are directly relevant to the job at hand. For example, if I the position I'm hiring for would benefit from a strong knowledge of a particular academic field, formal education in that field would be important... But, in general, I haven't found formal education to be very important for the types of programming that I do, so it's not important to me while I'm hiring.

Look, feel and other little things are important too. Not in the it-needs-to-look-pretty sense — I don't have any problem with a bland resume written in Word — but in the fanatic-attention-to-detail sense. I want a programmer who cares as much as I do about consistency, attention to detail and never, ever, under any circumstances using tabs and spaces interchangeably. If an applicant's resume has bullets that don't line up, text in random fonts and obviously missing punctuation, I'm going to assume that their code will look similar.

Cover letters. I have read one or two very good cover letters where the applicant gave specific reasons they were interested in the job and why they would be worth my time to interview. They were short, to the point, and distinctly lacking in phrases such as "I would make an excellent …". I appreciated these cover letters. I have skimmed and subsequently ignored all other cover letters.

tl;dr: I want a resume to show me in as much detail as is sensible what the applicant has done, what they know, and why they are a good programmer. Bonus points for interesting content and links to social sites like StackOverflow or Reddit.

PS: My startup is (or will in the near future be) hiring! If you're interested in working for a Toronto based startup which employs incredibly brilliant people, uses a lot of Python, produces a real, physical product, and pays real, actual money, send me an email: david@wolever.net.

The evils of `except:`

Thu, 29 Sep 2011 15:34:24 EDT

I had some discussion recently about the evils of using a naked except:. Here is a more complete description of the dangers, and of the correct solutions.

In short, except: is bad because hides the source source of the exception and frustrates debugging. For example, consider this code:

try:
    parsed = parse(file_name)
except:
    raise ParseError("can't parse file")

It will likely produce an error something like this:

$ ./my_program.py
Traceback (most recent call last):
    ...
ParseError: can't parse file

This kind of error makes me want to high-five someone. In the face. With a chair.

Notice that it does not contain any information about:

The file which caused the error
The line which caused the error
The nature of the error (is it an expected error? A bug? Who knows!)

And tracking down the source of this error would likely involve some binary searching on the input file or dropping into a debugger.

These are some other, equally unhelpful, bits of code that I have seen:

# There isn't much worse than completely hiding the error
except:
    pass

# Almost as bad is not giving any hit at what it was
except:
    print "there was an error!"

# And even showing the original error can be unhelpful if the error is
# something like an IndexError which could come from anywhere
except Exception, e:
    raise MyException("there was an error: %r" %(e, ))

Now, there is a situations where using a naked except: can be used safely. Exactly one.

1. The `except:` block is terminated with a `raise`

For example, when some cleanup needs to be done before leaving the function:

cxn = open_connection()
try:
    use_connection(cxn)
except:
    close_connection(cxn)
    raise

(note that, usually, the finally: block should be used for this kind of cleanup, but there are some situations where the code above makes more sense)

Every other situation should use except Exception, e::

2. A new exception is raised but the original stack trace is used

For example:

try:
    parsed = parse(file_name)
except Exception, e:
    raise ParseError("error parsing %r: %r" %(file_name, e)), None, sys.exc_info()[2]

A few things to note: first, the three expression version of raise is used, the third of which being the current stack trace. This means that the stack trace will point to the original source of the error:

File "my_program.py", line 9, in <module>
  parse(file_name)
File "parser.py", line 2, in parse
  for lineno, line in enumerate(open(file_name), "rb"):
ParseError: error parsing 'input.bin': TypeError("'str' object cannot be interpreted as an index",)

Instead of the (less helpful) line which re-raised the error:

File "my_program.py", line 11, in <module>
  raise ParseError("error parsing %r: %r" %(file_name, e))
ParseError: error parsing 'input.bin': TypeError("'str' object cannot be interpreted as an index",)

Second, the error includes the file name and original exception, which will make debugging significantly easier. When I'm writing particularly fragile code I'll often wrap the entire block in a try/except which will include as much state as is sensible in the error. For example, the main loop of the parse function might be:

def parse(file_name):
    lineno = -1
    current_foo = None
    try:
        f = open(file_name)
        for lineno, line in enumerate(f):
            current_foo = line.split()[0]
            ...
    except Exception, e:
        raise ParseError("error while parsing %r (line %r; current_foo: %r): %r"
                         %(file_name, lineno, current_foo, e)), None, sys.exc_info()[2]

3. The exception and stack trace are logged

For example, the main runloop of an application might be:

while 1:
    try:
        do_stuff()
    except Exception, e:
        log.exception("error in mainloop")
        time.sleep(1)

A few things to note: first, a naked except: should not be used here, as it will also catch KeyboardInterrupt and SystemExit exceptions, which is almost certainly a bad thing.

Second, log.exception is used, which includes a complete stack trace in the log (care should also be taken to make sure that these logs will be checked - for example by sending an email on exception logs).

Third, the time.sleep(1) ensures that the system won't get clobbered if the do_stuff() function immediately raises an exception.

Checking types in Python

Mon, 26 Sep 2011 13:53:08 EDT

A friend asked me recently when it's acceptable to check types in Python. Here is my reply:

It is almost never a good idea to check that function arguments are exactly the type you expect. For example, these two functions are very, very bad:

def add(a, b):
    if not isinstance(a, int):
        raise ValueError("a is not an int")
    if not isinstance(b, int):
        raise ValueError("b is not an int")
    return a + b

def sum(xs):
    if not isinstance(xs, list):
        raise ValueError("xs is not a list")
    base = 0
    for x in xs:
        base += x
    return base

There's no reason to impose those restrictions, and it makes life difficult if, for example, you want to add floats or sum an iterator:

>>> add(1.2, 3)
...
ValueError("a is not an int")
>>> sum(person.age for person in people)
...
ValueError("xs is not a list")

Type checking to correctly handle different kinds of input is occasionally acceptable, but should be used carefully (ex, to do optimizations, or situations where method overloading would be used in other languages). For example, these functions could be ok:

def contains_all(haystack, needles):
    if not isinstance(haystack, (set, dict)):
        haystack = set(haystack)
    return all(needle in haystack for needle in needles)

def ping_ip(addr):
    if isinstance(addr, numbers.Number):
        addr = numeric_ip_to_string(addr)
    # ping 'addr' which should be a string in "1.2.3.4" form
    ...

But it's almost always better to check for capabilities instead of checking for types. For example, if you want to make sure that add throws an error on invalid input, this would be a better way:

def add(a, b):
    if not (hasattr(a, "__add__") or hasattr(b, "__radd__")):
        raise ValueError("can't add a to b"))
    return a + b

This would be equivalent to excepting an interface instead of an implementation in a statically typed language:

// This is equivilent to ``isinstance(xs, list)`` -- usually bad
public static int sum(ArrayList xs) {
    ...
}

// This is equivilent to ``hasattr(xs, "__iter__")`` -- almost always better
public static int sum(Collection xs) {
    ...
}

Or better yet, Just Do It and wrap any exceptions which pop up:

def add(a, b):
    try:
        return a + b
    except Exception, e:
        raise ValueError("cannot add %r and %r: %r" %(a, b, e)), None, sys.exc_info()[2]

In general, though, code should assume that function arguments will behave correctly, then let the caller use your documentation and Python's helpful stack traces and debugging facilities to figure out what they did wrong.