Code Kills

In today's instalment of Adventures in Unicode, we meet the sneaky —.

When a web browser encounters —, it renders an em-dash (—). However, when — is decoded to Unicode (U+0097, 97₁₆ == 151₁₀), encoded to UTF-8 (\xc2\x97), written to a file, then opened with exactly the same web browser, the browser renders…

queue ominous music

Nothing!

Nothing is rendered because U+0097 is actually the END OF GUARDED AREA control character[0]… So it shouldn't be rendered.

So why is — being rendered? Because of our old friend, the Windows-1252 encoding, where character 151 is an em-dash. So the browser sees —, it helpfully assumes that the author is an idiot[1] and wanted an em-dash to be displayed instead of a control character[2].

What can be done?

I have been using a function which looks like this:

_fix_mixed_unicode_re = re.compile("([\x7F-\xFF]+)")
def fix_mixed_unicode(mixed_unicode):
    assert isinstance(mixed_unicode, unicode)
    def handle_match(match):
        return match.group(0).encode("raw_unicode_escape").decode("1252")
    return _fix_mixed_unicode_re.sub(handle_match, mixed_unicode)

It accepts a unicode string, then assumes any characters between 127 and 255 are actually Windows-1252 encoded, so it encodes them as bytes, then decodes those bytes as 1251, yielding a correct unicode string.

[0]: Which is represented by a line that looks very similar to an em-dash…

[1]: A generally safe assumption.

[2]: It should be noted that this happens regardless of the document's encoding.

After yesterdays post, Greg suggested I write another on how to test for Unicode safety... And unfortunately I've got some bad news: it's hard.

You never know when some developer, somewhere, will unintentionally encode or decode something the wrong way (for example, log("request for %s", unicode(url))).

But there is hope!

In my experience, almost all Unicode-related issues follow the same pattern: someone using str or unicode incorrectly and code which unexpectedly encodes/decodes a string.

The first is easy to check for: grep through the code for str( and unicode(.

The second is harder to check for, and requires an understanding of both the code base: all of the points where the code interacts with other parts of the system (filesystem, database, network) must be found and checked.

Finally, it isn't a bad idea to throw some Unicode into the test suite. Instead of calling mock users 'user0', 'user1', Call them u'\u03bcs\xeb\u044f' (u"μsëя")*. Keep a central "database" of these sorts of strings, so it's easy for developers who don't normally write in Cyrillic to use Cyrillic characters in their code (I keep my own personal list at http://wolever.net/~wolever/wiki/unicode_audit -- a url I can now type from memory).

One word of caution, though: you're asking for world of pain if you actually think you can commit UTF-8 encoded text -- any number of things will break (subversion may helpfully fail, your editor may helpfully re-encode the file, your unenlightened developers will complain about funny question marks in their code, etc...). Instead, have a central file which defines these "canned test strings" using escaped Python strings (ie, u'\u03bc...') then import that into your test suite:

from app.tests import i18n
...
def test_user():
    u = new User(name=i18n.user)
    ...

Or tar up all the offensive files, then write a script to un-tar them when they are needed**.

So, to sum it up:

• Make sure your developers grok (or, at least, understand) Unicode and encodings
• Make sure your code uses str and unicode safely
• Make sure your exit points are covered
• Make it really easy to include Unicode in tests

Then maybe, if you're lucky, all those inconsiderate people who have the audacity to ask for more than 127 characters will be able to use your application

*: A good choice both because it's easy for ignorant North Americans like myself to see that's it's correct.

**: This is how I tested DrProject's handling of Unicode filenames which are checked into the Subversion repository.

Unicode is an ugly beast... And until people start standardizing on Python 3k*, we're going to have to live with the eccentricities of Python 2's strings.

But, fear not! There is (at least some) hope. By changing a few patterns in the way you code, you can alleviate the bulk of Unicode-related problems.

First, using the str function. In just about every case, if you're using the str function, you're probably doing it wrong.

Let me demonstrate:

from hashlib import sha256
def hash(to_hash):
  hash = sha256(to_hash).hexdigest()
  print to_hash, ":", hash

Cool, we can hash things then print out the hash:

>>> hash('ohai')
ohai : e84712238709398f6d349dc2250b0efca4b72d8c2bfb7b74339d30ba94056b14

But, wait... What happens if the thing we're hashing isn't a string (even though it can be represented as a string):

>>> class Person:
...   name = 'David'
...   def __str__(self):
...     return self.name
...
>>> hash(Person())
...
TypeError: new() argument 1 must be string or read-only buffer, not instance

Oh no! Ok, let's fix the code:

from hashlib import sha256
def hash(to_hash):
  to_hash = str(to_hash) # Convert the object to a string before we hash it
  hash = sha256(to_hash).hexdigest()
  print to_hash, ":", hash

Great -- we can hash numbers now:

>>> hash(Person())
David : a6b54c20a7b96eeac1a911e6da3124a560fe6dc042ebf270e3676e7095b95652

And, for most people who only speak English, this is a perfect place to stop. After all, everything is a str, right?

Well... No. What happens if the input is a unicode object?

>>> p = Person()
>>> # This person had the audacity to give themselves a name containing
>>> # non-ascii symbols, so we represent it with a unicode object
>>> p.name = u'I\xf1t\xebrn\xe2ti\xf4n\xe0liz\xe6ti\xf8n'
>>> hash(p)
...
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position
1: ordinal not in range(128)

Crap. Where the heck is 'ascii' coming from?

Well, it's a long story (which I've covered over at Encoding and Decoding Text in Python), but basically the __str__ method of the unicode object (u'I\xf1...') is trying to encode the unicode object using the system's default encoding... Which, in this case, is ascii.

"Alright...", you're probably thinking, "If the problem is with unicode, maybe I could just replace that call to str with a call to unicode"

Ok, let's see what happens.

from hashlib import sha256
def hash(to_hash):
  to_hash = unicode(to_hash) # Convert the object to a unicode before we hash it
  hash = sha256(to_hash).hexdigest()
  print to_hash, ":", hash

>>> # this time, though, the person's name has come from The Internet, so it
>>> # is not yet a unicode object
>>> p.name = 'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti\xc3\xb4n\xc3\xa0liz\xc3\xa6ti\xc3\xb8n'
>>> hash(p)
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1:
ordinal not in range(128)

Yup, that's right -- you just can't win

What's happening here? Well, this time, the unicode function is trying to decode the input ('I\xc3...') into a unicode object... But, because the input isn't valid 7-bit ascii (again, the system's default), it explodes. Crap.

Confused yet?

So how can we save ourselves from all this insanity?

Actually, it's not too difficult:

1. Convert every single string, with out exception, to unicode as soon as they enter the system. For example, if you are writing a web application, GET and POST variables should be converted to unicode as soon as they are read from the environment:

   for (key, value) in environment.get_vars: request.GET[to_unicode(key)] = to_unicode(value)

2. Only convert the unicode objects back to str strings when you absolutly must. For example, when they are written to a file:

   log_file.write("New user '%s' created" %(to_str(p.name)))

3. Think hard before you call str or unicode. Each time your fingers type "s", "t", flashing lights and sirens should go off in your head, reminding you to make sure that the object you are string could never, ever, ever possibly contain unicode.

And what about those to_unicode and to_str functions? What should they look like? Well, probably something like this:

import locale
def to_unicode(text):
    """ Convert, at all consts, 'text' to a `unicode` object.

        Note: as a last-ditch effort, this function tries to decode the text
              as latin1... Which will always succeed.  If you expect to get
              text encoded with latin[2-9] or some other character set, this
              may not be desierable.

        >>> to_unicode(u'I\xf1t\xebrn\xe2ti')
        u'I\xf1t\xebrn\xe2ti'
        >>> to_unicode('I\xc3\xb1t\xc3\xabrn\xc3\xa2ti')
        u'I\xf1t\xebrn\xe2ti'
        >>> class Foo:
        ...   def __str__(self):
        ...       return 'foo'
        ...
        >>> f = Foo()
        >>> to_unicode(f)
        u'foo'
        >>> f.__unicode__ = u'bar'
        >>> to_unicode(f)
        u'bar'
        >>> """

    if isinstance(text, unicode):
        return text

    if hasattr(text, '__unicode__'):
        return text.__unicode__()

    text = str(text)

    try:
        return unicode(text, 'utf-8')
    except UnicodeError:
        pass

    try:
        return unicode(text, locale.getpreferredencoding())
    except UnicodeError:
        pass

    return unicode(text, 'latin1')


def to_str(text):
    """ Convert 'text' to a `str` object.

        >>> to_str(u'I\xf1t\xebrn\xe2ti')
        'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti'
        >>> to_str(42)
        '42'
        >>> to_str('ohai')
        'ohai'
        >>> class Foo:
        ...     def __str__(self):
        ...         return 'foo'
        ...
        >>> f = Foo()
        >>> to_str()
        'foo'
        >>> f.__unicode__ = lambda: u'I\xf1t\xebrn\xe2ti'
        >>> to_str(f)
        'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti'
        >>> """
    if isinstance(text, str):
        return text

    if hasattr(text, '__unicode__'):
        text = text.__unicode__()

    if hasattr(text, '__str__'):
        return text.__str__()

    return text.encode('utf-8')

So there you have it. A quick and fairly easy way to avoid many of your encoding-related options

If you're still not quite feeling comfortable with all of this, though, take a read over Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

*: My newly installed Debian 4 machine is still running Python 2.4 (released December 2004)... So that wait might be a while.