Code Kills

strftime: table of locale-aware formatters in different locales

2013-04-14T02:44:20Z

I got curious about what the different locale-specific strftime and strptime formatting directives produced in different locales… So I built a little script which would genreate a table showing each of the different locale-aware formatters in all of the different locales which I've got installed!

The code used to generate this table can be found at: bitbucket.org/wolever/locale-table.

	%a	%A	%b	%B	%x	%X	%p	%c
af_ZA	Tue	Tuesday	Aug	August	08/16/1988	21:30:05	PM	Tue Aug 16 21:30:05 1988
am_ET	ማክሰ	ማክሰኞ	ኦገስ	ኦገስት	16/08/1988	21:30:05	PM	ማክሰ ኦገስ 16 21:30:05 1988
be_BY	аў	аўторак	жні	жніўня	16.08.88	21:30:05	pm	аў 16 жні 21:30:05 1988
bg_BG	Вт	Вторник	Авг	Август	16.08.88	21:30:05	pm	Вт 16 Авг 21:30:05 1988
ca_ES	dim	dimarts	ago	agost	16/08/1988	21:30:05	PM	dim 16 ago 21:30:05 1988
cs_CZ	út	úterý	srp	srpna	1988/08/16	21:30:05	od	út 16 srp 21:30:05 1988
da_DK	Tir	Tirsdag	Aug	August	16.08.1988	21:30:05	pm	Tir 16 Aug 21:30:05 1988
de_AT	Di	Dienstag	Aug	August	16.08.1988	21:30:05	pm	Di 16 Aug 21:30:05 1988
de_CH	Di	Dienstag	Aug	August	16.08.1988	21:30:05	pm	Di 16 Aug 21:30:05 1988
de_DE	Di	Dienstag	Aug	August	16.08.1988	21:30:05	pm	Di 16 Aug 21:30:05 1988
el_GR	Τρι	Τρίτη	Αυγ	Αυγούστου	16/08/1988	21:30:05	μμ	Τρι 16 Αυγ 21:30:05 1988
en_AU	Tue	Tuesday	Aug	August	16/08/1988	21:30:05	pm	Tue 16 Aug 21:30:05 1988
en_CA	Tue	Tuesday	Aug	August	16/08/1988	21:30:05	pm	Tue 16 Aug 21:30:05 1988
en_GB	Tue	Tuesday	Aug	August	16/08/1988	21:30:05	pm	Tue 16 Aug 21:30:05 1988
en_IE	Tue	Tuesday	Aug	August	16/08/1988	21:30:05	pm	Tue 16 Aug 21:30:05 1988
en_NZ	Tue	Tuesday	Aug	August	16/08/1988	21:30:05	pm	Tue 16 Aug 21:30:05 1988
en_US	Tue	Tuesday	Aug	August	08/16/1988	21:30:05	PM	Tue Aug 16 21:30:05 1988
es_ES	mar	martes	ago	agosto	16/08/1988	21:30:05	PM	mar 16 ago 21:30:05 1988
et_EE	T	teisipäev	aug	august	16.08.1988	21:30:05		T, 16. aug 1988. 21:30:05
eu_ES	as.	asteartea	Abu	abuztua	1988/08/16	21:30:05	p.m.	1988 - Abu - 16 as. 21:30:05
fi_FI	Ti	Tiistai	Elo	Elokuu	16.08.1988	21:30:05	pm	Ti 16 Elo 21:30:05 1988
fr_BE	Mar	Mardi	aoû	août	16.08.1988	21:30:05		Mar 16 aoû 21:30:05 1988
fr_CA	Mar	Mardi	aoû	août	16.08.1988	21:30:05		Mar 16 aoû 21:30:05 1988
fr_CH	Mar	Mardi	aoû	août	16.08.1988	21:30:05		Mar 16 aoû 21:30:05 1988
fr_FR	Mar	Mardi	aoû	août	16.08.1988	21:30:05		Mar 16 aoû 21:30:05 1988
he_IL	ג'	שלישי	אוג	אוגוסט	16/08/88	21:30:05	PM	21:30:05 1988 אוג 16 ג'
hr_HR	Ut	Utorak	Kol	Kolovoz	16.08.1988	21:30:05	pm	Ut 16 Kol 21:30:05 1988
hu_HU	Ked	Kedd	Aug	Augusztus	1988/08/16	21:30:05	du	Ked Aug 16 21:30:05 1988
hy_AM	Երք	Երեքշաբթի	Օգս	Օգոստոս	16.08.1988	21:30:05		Երեքշաբթի, 16 Օգոստոս 1988 ի. 21:30:05
is_IS	þri	þriðjudagur	ágú	ágúst	16.08.1988	21:30:05	eh	þri 16 ágú 21:30:05 1988
it_CH	Mar	Martedì	Ago	Agosto	16.08.1988	21:30:05	pm	Mar 16 Ago 21:30:05 1988
it_IT	Mar	Martedì	Ago	Agosto	16.08.1988	21:30:05	pm	Mar 16 Ago 21:30:05 1988
ja_JP	火	火曜日	8	8月	1988/08/16	21時30分05秒	PM	火 8/16 21:30:05 1988
kk_KZ	сс	сейсенбі	там	тамыз	16.08.1988	21:30:05		сейсенбі, 16 тамыз 1988 ж. 21:30:05
ko_KR	화	화요일	8	8월	1988/08/16	21시 30분 05초	PM	화 8/16 21:30:05 1988
lt_LT	An	Antradienis	Rgp	rugpjūčio	1988.08.16	21:30:05		An Rgp 16 21:30:05 1988
nl_BE	di	dinsdag	aug	augustus	16-08-1988	21:30:05	pm	di 16 aug 21:30:05 1988
nl_NL	di	dinsdag	aug	augustus	16-08-1988	21:30:05	pm	di 16 aug 21:30:05 1988
no_NO	tir	tirsdag	aug	august	16.08.1988	21:30:05	pm	tir 16 aug 21:30:05 1988
pl_PL	wto	wtorek	sie	sierpnia	1988.08.16	21:30:05		wto 16 sie 21:30:05 1988
pt_BR	Ter	Terça Feira	Ago	Agosto	16/08/1988	21:30:05		Ter 16 Ago 21:30:05 1988
pt_PT	Ter	Terça Feira	Ago	Agosto	16.08.1988	21:30:05		Ter 16 Ago 21:30:05 1988
ro_RO	Mar	Marţi	Aug	August	16.08.1988	21:30:05	pm	Mar 16 Aug 1988 21:30:05
ru_RU	вт	вторник	авг	августа	16.08.1988	21:30:05		вторник, 16 августа 1988 г. 21:30:05
sk_SK	ut	utorok	aug	august	16.08.1988	21:30:05		ut 16 aug 21:30:05 1988
sl_SI	tor	torek	avg	avgust	16.08.1988	21:30:05	pm	tor 16 avg 21:30:05 1988
sr_YU	уто	уторак	авг	август	16.08.1988	21:30:05		уто 16 авг 21:30:05 1988
sv_SE	Tis	Tisdag	Aug	Augusti	16.08.1988	21:30:05	pm	Tis 16 Aug 21:30:05 1988
tr_TR	Sal	Salı	Ağu	Ağustos	16/08/1988	21:30:05	PM	Sal 16 Ağu 21:30:05 1988
uk_UA	вт	вівторок	сер	серпня	16.08.1988	21:30:05		вт 16 сер 21:30:05 1988
zh_CN	二	星期二	8	八月	1988/08/16	21时30分05秒	下午	二 8/16 21:30:05 1988
zh_HK	二	周二	8	8月	1988/08/16	21時30分05秒	下午	二 8/16 21:30:05 1988
zh_TW	二	周二	8	8月	1988/08/16	21時30分05秒	下午	二 8/16 21:30:05 1988

South's frozen models are big liars

2013-03-07T19:53:46Z

Consider South's orm object, http://south.readthedocs.org/en/latest/ormfreezing.html, which provides "frozen-in-time" versions of ORM models:

(Pdb++) from my_app.models import MyModel
(Pdb++) MyModel
<class 'my_app.models.MyModel'>
(Pdb++) orm['my_app.MyModel']
<class 'my_app.models.MyModel'>
(Pdb++) orm['my_app.MyModel'] == MyModel
False

Notice that the magic South frozen model appears to be in the same namespace as my real model: my_app.models!

This is at best stupid, and at worst potentially dangerous.

It makes perfect sense that South needs some mechanism for providing "frozen models" (ie, which match the database schema at the time of migration, of the current schema), but they are emphatically not the same as the real models (don't have any of the real methods, etc), so it should be made very clear that they are different! There are many ways this could be done, but one very simple step would be putting them in a different namespace, which makes it clear that these models are frozen, and references the migrations they belong to:

(Pdb++) orm['my_app.MyModel']
<class 'south.frozen_models.my_app.models_0003.MyModel'>

Additionally, this is a bad code smell which could potentially lead to bugs: because the "frozen model class" is lying about where it's from, any code which relies on the module path will do surprising things. In fact, it's such a big problem that pickle explicitly checks for this case:

(Pdb++) import pickle
(Pdb++) pickle.dumps(orm['my_app.MyModel'])
*** PicklingError: Can't pickle <class 'my_app.models.MyModel'>:
    it's not the same object as my_app.models.MyModel

Anti-Pattern: Duck typing with try/except

2012-11-29T23:46:19Z

An anti-pattern I've often seen is using try/except incorrectly in an attempt at duck typing:

try:
    widget.frob()
except AttibuteError:
    # widget can't be frobbed
    do_something_else()

This is fundamentally broken because it will conceal bugs in the widget.frob() method (ie, if there's a bug in widget.frob() which causes an AttibuteError to be raised).

In this situation, the only safe way to use try/except would be:

try:
    widget_frob = widget.frob
except AttibuteError:
    # widget can't be frobbed
    do_something_else()
else:
    widget_frob()

This ensures that bugs in widget.frob() won't be accidentally hidden.

Of course, this code is no where near as pretty as the "incorrect" version, which is why I've come to believe that using try/except for duck typing could be considered an anti-pattern, and that hasattr is generally preferable (except in situations where performance is important, as try/except will almost certainly be faster):

if hasattr(widget, "frob"):
    widget.frob()
else:
    # widget can't be frobbed
    do_something_else()

And of course, all this goes without saying that a naked except: is always an absolutely terrible idea...

The evils of `except:`

2011-09-29T19:34:24Z

I had some discussion recently about the evils of using a naked except:. Here is a more complete description of the dangers, and of the correct solutions.

In short, except: is bad because hides the source source of the exception and frustrates debugging. For example, consider this code:

try:
    parsed = parse(file_name)
except:
    raise ParseError("can't parse file")

It will likely produce an error something like this:

$ ./my_program.py
Traceback (most recent call last):
    ...
ParseError: can't parse file

This kind of error makes me want to high-five someone. In the face. With a chair.

Notice that it does not contain any information about:

The file which caused the error
The line which caused the error
The nature of the error (is it an expected error? A bug? Who knows!)

And tracking down the source of this error would likely involve some binary searching on the input file or dropping into a debugger.

These are some other, equally unhelpful, bits of code that I have seen:

# There isn't much worse than completely hiding the error
except:
    pass

# Almost as bad is not giving any hit at what it was
except:
    print "there was an error!"

# And even showing the original error can be unhelpful if the error is
# something like an IndexError which could come from anywhere
except Exception, e:
    raise MyException("there was an error: %r" %(e, ))

Now, there is a situations where using a naked except: can be used safely. Exactly one.

1. The `except:` block is terminated with a `raise`

For example, when some cleanup needs to be done before leaving the function:

cxn = open_connection()
try:
    use_connection(cxn)
except:
    close_connection(cxn)
    raise

(note that, usually, the finally: block should be used for this kind of cleanup, but there are some situations where the code above makes more sense)

Every other situation should use except Exception, e::

2. A new exception is raised but the original stack trace is used

For example:

try:
    parsed = parse(file_name)
except Exception, e:
    raise ParseError("error parsing %r: %r" %(file_name, e)), None, sys.exc_info()[2]

A few things to note: first, the three expression version of raise is used, the third of which being the current stack trace. This means that the stack trace will point to the original source of the error:

File "my_program.py", line 9, in <module>
  parse(file_name)
File "parser.py", line 2, in parse
  for lineno, line in enumerate(open(file_name), "rb"):
ParseError: error parsing 'input.bin': TypeError("'str' object cannot be interpreted as an index",)

Instead of the (less helpful) line which re-raised the error:

File "my_program.py", line 11, in <module>
  raise ParseError("error parsing %r: %r" %(file_name, e))
ParseError: error parsing 'input.bin': TypeError("'str' object cannot be interpreted as an index",)

Second, the error includes the file name and original exception, which will make debugging significantly easier. When I'm writing particularly fragile code I'll often wrap the entire block in a try/except which will include as much state as is sensible in the error. For example, the main loop of the parse function might be:

def parse(file_name):
    lineno = -1
    current_foo = None
    try:
        f = open(file_name)
        for lineno, line in enumerate(f):
            current_foo = line.split()[0]
            ...
    except Exception, e:
        raise ParseError("error while parsing %r (line %r; current_foo: %r): %r"
                         %(file_name, lineno, current_foo, e)), None, sys.exc_info()[2]

3. The exception and stack trace are logged

For example, the main runloop of an application might be:

while 1:
    try:
        do_stuff()
    except Exception, e:
        log.exception("error in mainloop")
        time.sleep(1)

A few things to note: first, a naked except: should not be used here, as it will also catch KeyboardInterrupt and SystemExit exceptions, which is almost certainly a bad thing.

Second, log.exception is used, which includes a complete stack trace in the log (care should also be taken to make sure that these logs will be checked - for example by sending an email on exception logs).

Third, the time.sleep(1) ensures that the system won't get clobbered if the do_stuff() function immediately raises an exception.

Checking types in Python

2011-09-26T17:53:08Z

A friend asked me recently when it's acceptable to check types in Python. Here is my reply:

It is almost never a good idea to check that function arguments are exactly the type you expect. For example, these two functions are very, very bad:

def add(a, b):
    if not isinstance(a, int):
        raise ValueError("a is not an int")
    if not isinstance(b, int):
        raise ValueError("b is not an int")
    return a + b

def sum(xs):
    if not isinstance(xs, list):
        raise ValueError("xs is not a list")
    base = 0
    for x in xs:
        base += x
    return base

There's no reason to impose those restrictions, and it makes life difficult if, for example, you want to add floats or sum an iterator:

>>> add(1.2, 3)
...
ValueError("a is not an int")
>>> sum(person.age for person in people)
...
ValueError("xs is not a list")

Type checking to correctly handle different kinds of input is occasionally acceptable, but should be used carefully (ex, to do optimizations, or situations where method overloading would be used in other languages). For example, these functions could be ok:

def contains_all(haystack, needles):
    if not isinstance(haystack, (set, dict)):
        haystack = set(haystack)
    return all(needle in haystack for needle in needles)

def ping_ip(addr):
    if isinstance(addr, numbers.Number):
        addr = numeric_ip_to_string(addr)
    # ping 'addr' which should be a string in "1.2.3.4" form
    ...

But it's almost always better to check for capabilities instead of checking for types. For example, if you want to make sure that add throws an error on invalid input, this would be a better way:

def add(a, b):
    if not (hasattr(a, "__add__") or hasattr(b, "__radd__")):
        raise ValueError("can't add a to b"))
    return a + b

This would be equivalent to excepting an interface instead of an implementation in a statically typed language:

// This is equivilent to ``isinstance(xs, list)`` -- usually bad
public static int sum(ArrayList xs) {
    ...
}

// This is equivilent to ``hasattr(xs, "__iter__")`` -- almost always better
public static int sum(Collection xs) {
    ...
}

Or better yet, Just Do It and wrap any exceptions which pop up:

def add(a, b):
    try:
        return a + b
    except Exception, e:
        raise ValueError("cannot add %r and %r: %r" %(a, b, e)), None, sys.exc_info()[2]

In general, though, code should assume that function arguments will behave correctly, then let the caller use your documentation and Python's helpful stack traces and debugging facilities to figure out what they did wrong.

Python 2.X's str.format is unsafe

2011-09-22T23:33:00Z

I posted a tweet today when I learned that Python's %-string-formatting isn't actually a special case - the str class just implements the __mod__ method.

One side effect of this is that a few people commented that %-formatting is to be replaced with .format formatting... So I'd like to take this opportunity to explain why .format string formatting is unsafe in Python 2.X.

With %-formatting, if the format string is a str while one of the replacements is a unicode the result will be unicode:

>>> "Hello %s" %(u"world", )
u'Hello world'

However, .format will always return the same type of string (str or unicode) as the format string:

>>> "Hello {}".format(u"world")
'Hello world'

This is a problem in Python 2.X because unqualified string literals are instances of str, and the implicit encoding of unicode arguments will almost certainly explode at the least opportune moments:

>>> "Hello {}".format(u"\u263a")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u263a' in position 0: ordinal not in range(128)

Of course, one possible solution to this is remembering to prefix all string literals with u:

>>> u"Hello {}".format(u"\u263a")
u'Hello \u263a'

But I prefer to simply use %-style formatting, because then I don't need to remember anything:

>>> "Hello %s" %(u"\u263a", )
u'Hello \u263a'
>>> print _.encode('utf-8')
Hello ☺

Of course, as you've probably noticed, this means that the format string is being implicitly decoded to unicode... But since my string literals generally don't contain non-ASCII characters it's not much of an issue.

Note that this is not a problem in Py 3k because string literals are unicode.

Lies, More Lies and Python Packaging Documentation on `package_data`

2011-07-16T00:35:18Z

My slice of Python packaging hell today was thanks to the lie that is package_data.

You see, I've been trying to create an package that includes non-Python files in the distribution... So I did what any good developer would do and hit the documentation:

Package data can be added to packages using the package_data keyword argument to the setup() function.

—Distutils documentation

and

If you want finer-grained control over what files are included (for example, if you have documentation files in your package directories and want to exclude them from installation), then you can also use the package_data keyword.

—Distribute documentation

Over the last hour, though, I've learned that these statements are somewhere between “dangerously misleading” and “damn lies”.

This is because the primary type of Python package is a source package, and the canonical method for creating a source package is by using setup.py sdist. However, the data specified in package_data are not included in source distributions — they are only included in binary (setup.py bdist) distributions and installs (setup.py install).

The only way to get package data included in source packages is the MANIFEST.in file... Which will also include data in binary distributions and installs.

Which renders the package_data option useful only if sdist is not used… And dangerously misleading if sdist is used.

tl;dr: package_data is a lie. Ignore it. Only use MANIFEST.in.

Tips for Managing a Django Project

2011-06-22T20:44:05Z

During the time I've spent with Django, I've picked up a couple tricks for making life a little bit less terrible.

First, split the project project into three (or more) parts, each with its own settings.py: the main project, the development environment and the production environment. For example, my current project, code named eos, has a directory structure something like this:

eos/
    .hg/
    .hgignore
    manage.py
    run
    eos/
        __init__.py
        settings.py
        templates/
        urls.py
        ...
    hacking/
        __init__.py
        settings.py
        db.sqlite3
        ...
    production/
        __init__.py
        settings.py
        run.wsgi
    ...

The eos/ directory is more or less a standard Django project (ie, created by django-admin.py startproject), except that eos/settings.py does not have any environment-specific information in it (for example, it doesn't have any database settings or URLs).

The hacking/ and production/ directories also contain settings.py files, except they define only environment specific settings. For example, hacking/settings.py looks a bit like this:

from eos.settings import *
path = lambda *parts: os.path.join(os.path.dirname(__file__), *parts)

DATABASE_ENGINE = "sqlite3"
DATABASE_NAME = path("db.sqlite3")

DEBUG = True

While production/settings.py contains:

from eos.settings import *

DATABASE_ENGINE = "psycopg2"
DATABASE_NAME = "eos"
DATABASE_USER = "eos"
DATABASE_PASSWORD = "secret"

DEBUG = False

Then, instead of configuring Django (ie, calling setup_environment) on eos.settings, it is called on either hacking.settings or production.settings. For example, manage.py contains:

...
import hacking.settings
execute_manager(hacking.settings)

And production/run.wsgi contains:

...
os.environ["DJANGO_SETTINGS_MODULE"] = "production.settings"
...

Second, every settings.py file should contain the path lambda:

path = lambda *parts: os.path.join(os.path.dirname(__file__), *parts)

It will make specifying paths relative to the settings.py file very easy, and completely do away with relative-path-related issues. For example:

MEDIA_ROOT = path("media/")
DATA_ROOT = path("data/")
DATABASE_NAME = path("db.sqlite3")

Third, there should be scripts for running, saving and re-building the environment. I use two scripts for this: run and dump_dev_data. By default the run script calls ./manage.py runserver 8631 (specifying a port is useful so that web browsers can distinguish between different applications - keeping passwords, history, etc. separate). Run can also be passed a reset argument, which will delete the development database and rebuild it from the dev fixtures. These fixtures are created by the dump_dev_data script, which calls ./manage.py dumpdata for each application, saving the data to fixtures named dev (these fixtures are committed along side the code, so all developers can work off the same data).

So, for example, when I'm developing a new model, my workflow will look something like this:

... add new model to models.py ...
$ ./run reset # Reset the database adding the new model
... use the website to create data for the new model ...
$ ./dump_dev_data # Dump the newly created data
$ hg commit -m "Adding new model + test data"

Python security tip: urllib/urllib2 will read `file://` URLs

2011-06-06T03:03:00Z

I discovered, entirely by accident, that urllib2.urlopen, urllib.urlretrieve, and probably others, will happily read file:// urls and filesystem paths. For example:

>>> import urllib, urllib2
>>> urllib.urlretrieve("database_connection_settings.txt", "/tmp/temp_file")
('/tmp/temp_file', <mimetools.Message instance at 0x…>)
>>> urllib2.urlopen("file:///dev/urandom").read(10)
'\xf1r?\x0fC\x86p\x05\xa4\xdd'

This means that applications which blindly urlopen untrusted URLs (for example, from RSS feeds) are potentially vulnerable to information disclosure and denial of service attacks.

Bug caused by Python's significant whitespace

2011-05-25T19:12:00Z

I encountered my first bug (in recent memory) that was caused by Python's significant whitespace today.

As I was editing a class, I accidentally left-shifted an entire method, effectively removing the rest of the methods from the class:

class Foo:
    def foo(self):
        …

def accidentally_shifted_left(self):
    …

    def bar(self):
        …