Code Kills - Python

Absolute paths must die

david@wolever.net (David Wolever) — Sat, 13 Feb 2010 17:23:22 -0500

Ok guys, it's pretty simple: if you're working in a Turing-complete environment and you've got paths which start with a '/', you're doing it wrong. Very wrong!

Todays offender: WSGI scripts.

Here's the first line of a common WSGI script:

app_path = '/path/to/application'

See a problem there? That's right, the absolute path!

How could this be solved? Simple: put the WSGI script in /path/to/application, then get the app_path using: app_path=os.path.abspath(os.path.dirname(__file__)). Problem solved.

</rant>

Django settings.py mini-tip: the `path` lambda

david@wolever.net (David Wolever) — Sun, 17 Jan 2010 22:26:41 -0500

There are a few things I do every time I start a new Django site, and one of those things is add my path lambda to the top of settings.py:

    import os
    ROOT = os.path.abspath(os.path.dirname(__file__))
    path = lambda *args: os.path.join(ROOT, *args)

From then on, it's drop-dead simple to create absolute paths which are relative to the root of the project (or, at least, the directory which contains settings.py). For example:

    >>> path('media/')
    '/Users/wolever/code/review_anywhere/media/

And some other places where it is useful:

    DATABASE_ENGINE = 'sqlite3'
    DATABASE_NAME = path('db.sqlite3')
    ...
    TEMPLATE_DIRS = (
        path('templates/'),
    )
    ...
    DATA_DIR = path('data/')
    ...

str(...): 'yer probably doin' it wrong.

david@wolever.net (David Wolever) — Tue, 10 Feb 2009 11:13:19 -0500

Unicode is an ugly beast... And until people start standardizing on Python 3k*, we're going to have to live with the eccentricities of Python 2's strings.

But, fear not! There is (at least some) hope. By changing a few patterns in the way you code, you can alleviate the bulk of Unicode-related problems.

First, using the str function. In just about every case, if you're using the str function, you're probably doing it wrong.

Let me demonstrate:

from hashlib import sha256
def hash(to_hash):
  hash = sha256(to_hash).hexdigest()
  print to_hash, ":", hash

Cool, we can hash things then print out the hash:

>>> hash('ohai')
ohai : e84712238709398f6d349dc2250b0efca4b72d8c2bfb7b74339d30ba94056b14

But, wait... What happens if the thing we're hashing isn't a string (even though it can be represented as a string):

>>> class Person:
...   name = 'David'
...   def __str__(self):
...     return self.name
...
>>> hash(Person())
...
TypeError: new() argument 1 must be string or read-only buffer, not instance

Oh no! Ok, let's fix the code:

from hashlib import sha256
def hash(to_hash):
  to_hash = str(to_hash) # Convert the object to a string before we hash it
  hash = sha256(to_hash).hexdigest()
  print to_hash, ":", hash

Great -- we can hash numbers now:

>>> hash(Person())
David : a6b54c20a7b96eeac1a911e6da3124a560fe6dc042ebf270e3676e7095b95652

And, for most people who only speak English, this is a perfect place to stop. After all, everything is a str, right?

Well... No. What happens if the input is a unicode object?

>>> p = Person()
>>> # This person had the audacity to give themselves a name containing
>>> # non-ascii symbols, so we represent it with a unicode object
>>> p.name = u'I\xf1t\xebrn\xe2ti\xf4n\xe0liz\xe6ti\xf8n'
>>> hash(p)
...
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position
1: ordinal not in range(128)

Crap. Where the heck is 'ascii' coming from?

Well, it's a long story (which I've covered over at Encoding and Decoding Text in Python), but basically the __str__ method of the unicode object (u'I\xf1...') is trying to encode the unicode object using the system's default encoding... Which, in this case, is ascii.

"Alright...", you're probably thinking, "If the problem is with unicode, maybe I could just replace that call to str with a call to unicode"

Ok, let's see what happens.

from hashlib import sha256
def hash(to_hash):
  to_hash = unicode(to_hash) # Convert the object to a unicode before we hash it
  hash = sha256(to_hash).hexdigest()
  print to_hash, ":", hash

>>> # this time, though, the person's name has come from The Internet, so it
>>> # is not yet a unicode object
>>> p.name = 'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti\xc3\xb4n\xc3\xa0liz\xc3\xa6ti\xc3\xb8n'
>>> hash(p)
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1:
ordinal not in range(128)

Yup, that's right -- you just can't win

What's happening here? Well, this time, the unicode function is trying to decode the input ('I\xc3...') into a unicode object... But, because the input isn't valid 7-bit ascii (again, the system's default), it explodes. Crap.

Confused yet?

So how can we save ourselves from all this insanity?

Actually, it's not too difficult:

Convert every single string, with out exception, to unicode as soon as they enter the system. For example, if you are writing a web application, GET and POST variables should be converted to unicode as soon as they are read from the environment:

for (key, value) in environment.get_vars: request.GET[to_unicode(key)] = to_unicode(value)
Only convert the unicode objects back to str strings when you absolutly must. For example, when they are written to a file:

log_file.write("New user '%s' created" %(to_str(p.name)))
Think hard before you call str or unicode. Each time your fingers type "s", "t", flashing lights and sirens should go off in your head, reminding you to make sure that the object you are string could never, ever, ever possibly contain unicode.

And what about those to_unicode and to_str functions? What should they look like? Well, probably something like this:

import locale
def to_unicode(text):
    """ Convert, at all consts, 'text' to a `unicode` object.

        Note: as a last-ditch effort, this function tries to decode the text
              as latin1... Which will always succeed.  If you expect to get
              text encoded with latin[2-9] or some other character set, this
              may not be desierable.

        >>> to_unicode(u'I\xf1t\xebrn\xe2ti')
        u'I\xf1t\xebrn\xe2ti'
        >>> to_unicode('I\xc3\xb1t\xc3\xabrn\xc3\xa2ti')
        u'I\xf1t\xebrn\xe2ti'
        >>> class Foo:
        ...   def __str__(self):
        ...       return 'foo'
        ...
        >>> f = Foo()
        >>> to_unicode(f)
        u'foo'
        >>> f.__unicode__ = u'bar'
        >>> to_unicode(f)
        u'bar'
        >>> """

    if isinstance(text, unicode):
        return text

    if hasattr(text, '__unicode__'):
        return text.__unicode__()

    text = str(text)

    try:
        return unicode(text, 'utf-8')
    except UnicodeError:
        pass

    try:
        return unicode(text, locale.getpreferredencoding())
    except UnicodeError:
        pass

    return unicode(text, 'latin1')


def to_str(text):
    """ Convert 'text' to a `str` object.

        >>> to_str(u'I\xf1t\xebrn\xe2ti')
        'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti'
        >>> to_str(42)
        '42'
        >>> to_str('ohai')
        'ohai'
        >>> class Foo:
        ...     def __str__(self):
        ...         return 'foo'
        ...
        >>> f = Foo()
        >>> to_str()
        'foo'
        >>> f.__unicode__ = lambda: u'I\xf1t\xebrn\xe2ti'
        >>> to_str(f)
        'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti'
        >>> """
    if isinstance(text, str):
        return text

    if hasattr(text, '__unicode__'):
        text = text.__unicode__()

    if hasattr(text, '__str__'):
        return text.__str__()

    return text.encode('utf-8')

So there you have it. A quick and fairly easy way to avoid many of your encoding-related options

If you're still not quite feeling comfortable with all of this, though, take a read over Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

*: My newly installed Debian 4 machine is still running Python 2.4 (released December 2004)... So that wait might be a while.

Python Brain-Teaser

david@wolever.net (David Wolever) — Thu, 29 May 2008 11:14:57 -0400

I'm working on pulling some functionality out of one object and putting it in another, and I came across this interesting problem:

class Foo:

me = "foo"

class Bar:

me = "bar"

def get_me(self):

return self.me

Foo.get_me = Bar.get_me

x = Foo()

print x.get_me()

What does this print?

And, next question, why is that?

After lunch I'll post my thoughts

Continue reading "Python Brain-Teaser"

SSH and HTTPS on the same port?!

david@wolever.net (David Wolever) — Fri, 11 Apr 2008 17:38:16 -0400

A friend of mine was over the other night, and we were talking about how much we like running SSH on port 443. It gets you around all but the most insane firewalls, and to anyone but the keenest of observers it just looks like HTTPS traffic.

But running SSH over port 443 makes it particularly difficult to run a secure web server as well.

So what can be done?

Well, it turns out that when an SSH client initiates a connection, it waits for the server to send a banner (SSH-1.99-OpenSSH_4.7, for example) before beginning the secure negotiation. Browsers, on the other hand, just go right ahead and begin the secure negotiation.

Knowing this, it didn't take long to formulate a plan: write a little program which will listen on 443. When it gets a connection, it figures out if the remote end is trying to speak SSH or HTTPS, makes the appropriate connection on the local end, the passes the data between the two.

It only took a couple of hours to write a small proof-of-concept... And, believe it or not, it even works! Disclaimer: this particular implementation is not quite suitable for any real use.

Anyway, this may not be the most efficient way of doing things... But that's no matter. It's still neat

Code: sstp.py

Reverting Changes in SVN (or: it's not as easy as svn up -r)

david@wolever.net (David Wolever) — Fri, 08 Feb 2008 11:57:24 -0500

If there is one thing that I've found universally confusing about version control systems, it's how to revert back to a previous revision, then move on from there. Conventional wisdom would dictate that svn up -r $OLD with some additional flag that says pretend that we're still at HEAD would do the trick... But, alas, there exists no such flag.

So, besides svn cat -r $OLD $FILE > $FILE, what's the best way to revert a file (or entire tree) to an older revision? Well, it turns out that merge is the tool you're looking for.

The trick is that you can diff backwards as well as forwards:

[wolever@thebes] ~/test_dr/All svn diff -r 2:1
Index: stuff
===================================================================
--- stuff       (revision 2)
+++ stuff       (revision 1)
@@ -1,4 +1,3 @@
 Let us endeavor so to live that when we come to die even the undertaker will be
 sorry.
                -- Mark Twain, "Pudd'nhead Wilson's Calendar"
-Increased knowledge will help you now.  Have mate's phone bugged.
Index: docs.html

Which means that merge will work both ways as well:

[wolever@thebes] ~/test_dr/All svn merge -r 2:1 stuff
U    stuff
[wolever@thebes] ~/test_dr/All svn ci -m "reverted changes" stuff
[wolever@thebes] ~/test_dr/All svn diff -r 1:3 stuff
[wolever@thebes] ~/test_dr/All

Of course, this will also work with the myriad of other version control tools out there. In fact, if you're still using SVN, reverting old changes is probably the least of your problems... At least compared to, say, merging different branches ^_^

But that's a post for another day -- I need to stop writing about merging and actually get on with, err, doing it...

Amazon S3 (or, How I Solved My Image Hosting Woes)

david@wolever.net (David Wolever) — Tue, 12 Jun 2007 12:19:42 -0400

A couple months ago, I read Jeff Atwood's article on Using Amazon S3 as an Image Hosting Service, but because my favicon does not does not generate 27 GBs of traffic a month, I didn't think much of it.

(if you're scratching your head wondering what Amazon's S3 is, I'll do my best to explain. It stands for Simple Storage Service, and it is just that: a simple service that allows you to store data. That data (or "those files", if you wish) can be world-readable (from a web browser) or private (many people are using it for backup). It's targeted at developers (Amazon only provides a set of APIs -- all GUIs are 3rd party) and it is dirt cheep: only $0.15 a gigabyte.)

Then, a few weeks ago, I was taking some pictures of my sister's soccer practice, and realized that I needed some place to put them. After looking at a couple of cheep hosting plans, I decided that I didn't like them; $5 a month isn't much, but that's still $60 a year for a whole lot of space I won't be using. Then I remembered S3.

After a night of playing around, I figured that it should be pretty simple. Write a script that will go through an HTML file looking for images, then upload them one at a time, replacing the links as it goes. html2s3 was born.

It turns out that the S3 library provided by Amazon is very easy to use, with most of my time spent correcting file paths. A couple of neat bits:

parser = OptionParser(usage = "usage: %prog [options] [FILES]\nWill process ..."
parser.add_option("-b", "--bucket", action="store", dest="bucket", help=...)
parser.add_option("-k", "--key-prefix", dest="remote_base_path", help="Prefix key ...")
parser.add_option("-n", "--no-backup", dest="no_backup", action="store_true" ...)
(options, args) = parser.parse_args()

I had seen OptionParser before, but never used it. It's so simple, though, that I think I'm going to use it quite a bit more frequently.

source = args or sys.stdin.xreadlines()
for line in source:
process_entry(line)

One thing I loved about C was the ability to use assignment in loops (while (fgets(buf, sizeof(buf), stdin) != EOF)), but stdin.xreadlines() does the same sort of thing.

Update: When I moved the script over to my server, it started telling me that I didn't have permission to list the bucket. So I started digging in to it a bit, and found out that I was getting the error RequestTimeTooSkewed. It turns out that I forgot to turn my server's clock back last daylight savings... D'oh!

Debugging Python

david@wolever.net (David Wolever) — Tue, 29 May 2007 14:36:26 -0400

There are two modules that I use to debug Python code: pdb and rpdb2.

`pdb`

There are two main ways that I launch pdb: from the command line and from a breakpoint within a script.

To launch pdb from the command line, simply run Python with the -m pdb flag:

# py -m pdb script name

(if you're like me, python is a bit much to type... A simple fix is to just create a link to the python binary named py: ln -s $(which python) /usr/bin/py)

The second, and much more useful, method of invoking pdb is the set_trace function. To break at any point in your code, simply insert the line:

import pdb; pdb.set_trace()

When the code execution gets to that point, it will stop and drop you in to a debugging shell.
Just to feel cool like the Eclipse kids, I have mapped this to F8 in Vim (map <F8> Oimport pdb; pdb.set_trace() #BREAK<Esc>)

`rpdb2`

rpdb2 stands for Remote Python Debugger 2, and as the name suggests it allows remote debugging. This is useful if pdb does not play well with the threading/forking in your application (it did not like web.py, for example). The other, and much greater, benefit of rpdb2 is Winpdb. As the name suggests, Winpdb is a graphical Python debugger.

While Winpdb allows you to run your script from within the application, that is not very helpful if you are working on a beast like Drproject.

When you're working with a beastly application, it is much easier to set a rpdb2 breakpoint and then attach Winpdb to the application. To do this, stick this line where ever you want your application to break:

import rpdb2; rpdb2.start_embedded_debugger("asdf")

When your application gets to that line, it will "hang" as the debugger runs. Now it's time to fire up Winpdb and attach (File --> Attach) to the process. The password is "asdf". After it loads, you will be able to do debug the application like normal.

One final note about using rpdb2: to run a Python command (for instance, in the bottom right window of Winpdb), you have to prefix it with exec (for instance, exec print foo).