Code Kills

I've got some code checked out with git-svn, and I'd like to do two things: pull in new revisions and push up my local revisions. Pretty simple, brainless, thing, right? But we're dealing with git, so of course not

# Fortunately I'm smart enough to remember that when you
# want to push changes to SVN, you've obviously got to use: 
$ git-svn dcommit
...
$
# Hurra! It worked! (at least I think...)
# Now for the next feat, pulling in more revisions...
$ git svn
...
  fetch            Download new revisions from SVN
...
# Phew! That looks like exactly what I need... Maybe it's not so tricky after all!
$ git svn fetch
        A       basie/a3c/admin.py
r13 = 45f5309121a77f33f8bd87009671727c0e2dc4a5 (zuze)
# Sweet! Now I can take a look at what has been changed
$ cat basie/a3c/admin.py
cat: basie/a3c/admin.py: No such file or directory
# Hu? I thought I just fetched it...
# Crap, right! I'm NOT supposed to use fetch, I'm supposed to use 'rebase'
# (not, of course, that I understand why "pulling in new revisions from SVN
#  is equivilent to a mung-your-history-rebase... But that's ok, I guess)
$ git svn rebase
...
$ cat basie/a3c/admin.py
cat: basie/a3c/admin.py: No such file or directory
# Blast.  That didn't work either.
# Right, that's because 'fetch' updates the git repository but not the
# working tree... Alright, let's try an update
$ git up
git: 'up' is not a git-command. See 'git --help'.
# Crap, right, git is too cool to have 'update'... I think I need reset
$ git reset
docs/cmdline.wiki: needs update
# hhmm... Why is it telling me that the files I edited need update?
# And how on earth do I update them?
# I give up  Can someone smarter than I am tell me what to do?

I guess I'll go back to Bazaar... It may be slow as molasses on a cold day, but at least it is simple enough that mere mortals can use it without resorting to Google.

I'm working on pulling some functionality out of one object and putting it in another, and I came across this interesting problem:

What does this print?

And, next question, why is that?

After lunch I'll post my thoughts

This tip was originally posted to the python-dev mailing list, so I can't take one scrap of credit for it.

Basically, since OpenSSH4, there has been an option to share connections -- that is, once you've opened one connection to a host, every subsequent connection is tunneled through the same channel, completely removing the overhead of authentication!

It's quite simple, just add this to ~/.ssh/config:

ControlMaster auto
ControlPath ~/.ssh/.%r@%h:%p

There can be problems if your machine crashes and that file is left lying around... So adding this to your crontab will fix that:

@reboot rm -f .ssh/controls/*

Next time: tab completion with scp!

I was not impressed when I tried to check out a UTF-8 encoded file with SVK, then got the helpful message Can't encode path as ascii:

$ svk up
Syncing //drp/trunk(/drp/trunk) in /home/wolever/Trunk to 5388.
Can't encode path as ascii.

I was even less impressed when I searched Google for svk unicode and this blog was the first hit.

Fortunately my Google-foo is high today, and I was able to find a page that gives a solution: Making sure that your locale is set to something similar to en_US.UTF-8.

On Debian, here's how I do it:

$ sudo apt-get install locales # Ubuntu 6.06 didn't have it...
...
$ export LANG="en_US.UTF-8"
$ export LANGUAGE="$LANG"

And, of course, it may be good to put those two exports in ~/.bashrc.

Oh, but wait!

$ svk up
Syncing //drp/trunk(/drp/trunk) in /home/wolever/Trunk to 5388.
Can't encode path as ascii.

It still doesn't work!

It turns out that, for what ever reason, something was upset. Eventually I got it working by deleting the offending directory, then using svk revert to revert it:

$ rm -r hacking/
$ svk revert -R hacking
...
Reverted hacking/utf8_爱的

Hurra!

When dealing with Unicode in Python, it doesn't take long to get the dreaded 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128).

You never see it coming. It doesn't make any sense. You didn't even ask for ascii!

So what's the deal?

I'm glad you asked. I will demonstrate:

>>> s = file("data").read()
>>> s
'SGVsbG8sIHdvcmxkIQ==\n'

If you guessed that s is a hunk of base64 encoded data, you'd be right! Give yourself a gold star.

Now, if we want to do anything useful with this data, it needs to be decoded:

>>> s.decode('base64')
'Hello, world!'

We have just taken an encoded hunk of data and decoded it to get a useful hunk of data.

>>> s.decode('base64').replace('world', 'Marguerite')
'Hello, Marguerite!'
>>> _.encode('base64')
'SGVsbG8sIE1hcmd1ZXJpdGUh\n'

Now we can take that useful hunk of data (the English in 7-bit ASCII), do something useful with it (in this case, replace 'world' with 'Marguerite'), and finally encode the data.

So how does all this relate back to Unicode and ascii error messages?

I have used base64 encoded data here, but the same concept applies when dealing with Unicode data:

1. Hunk of opaque data comes in (but we know that it contains some sort of Unicode text)
2. Hunk of opaque data is decoded, creating a unicode object
3. The unicode object is used for something useful
4. The unicode object is encoded and saved (to disk, to a database, or sent to a browser)

(of course, in the Real World, you've got to figure out which encoding was used on the data (UTF-8, Latin1, etc)... But that's a topic for another post.)

Ok, back to the 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128) error. It should be fairly clear that this error is coming up because Python is trying to decode a bunch of bytes as 7-bit ASCII, but some of them are out of that range (eg, they have a value over 127).

I know what you're saying, "but I never asked Python to decode anything! I'm just trying to turn it into unicode!"

>>> unicode("Ol\xc3\xa1, mundo!")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)

Two questions arise here: First, "Where is the 'ascii' coming from?" Second, "How do I make it work?"

To answer the first question, it's important to think about what's happening when the call to unicode(...) is made. The unicode function accepts an encoded string, decodes it, and creates a unicode object. In this case, though, we haven't given the function any indication of which decoder it should use, so it falls back to the computer's default encoding: ascii.

So how can you make it work? Tell unicode which encoding to use:

>>> unicode("Ol\xc3\xa1, mundo!", 'utf8')
u'Ol\xe1, mundo!'

(now, as I mentioned before, figuring out which encoding to use is another huge problem... But I'll leave that for another day)

Another problem I run into quite often is this:

>>> "Ol\xc3\xa1, mundo!".encode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)

And, by now, the cause of this should be painfully obvious: I've given Python an encoded string, so I should be decoding it, not encoding it again.

But why the confusing error message? Well, I'm not entirely sure, but my guess is that the UTF-8 encoder expects a unicode object, so it tries to convert the input (in this case, "Ol\xc3...") to Unicode before encoding it.

Is there any end to this insanity?!

Yes! Python 3000 will have two distinct classes: one for strings, one for hunks of data. Whenever data is read, it will come in as a "hunk of data". It will have to be explicitly decoded to a string before it can be used as such. Hopefully that will make life a little bit less painful.

See also:

• The obligatory link to Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets
• PEP 3112 - Byte literals in Python 3000

I got a fairly significant patch to DrProject today. I wanted to show it to the other developers so they could give it a look over. How'd I do it?

    $ patch -p0 < patch
    ... check that it works ...
    # Create a new branch for the patch
    $ svk cp //drp/trunk //drp/branches/patch -m "Created branch for patch"
    # Do an in-place switch of my repository (which takes about 2 seconds,
    # versus the minute or two it would take to check the entire tree out)
    $ svk switch //drp/branches/patch 
    # Because switch doesn't revert modified files, the changes are still here,
    # ready to be checked in
    $ svk ci -m "Checking in patched code"

Now they can use DrProject's fancy online changeset viewer, checkout the code for themselves, use svn diff... The list is endless.

Now that rocks SO much more than mailing around .diff files (and makes life happier when it comes time to pull it back into trunk).

A friend of mine was over the other night, and we were talking about how much we like running SSH on port 443. It gets you around all but the most insane firewalls, and to anyone but the keenest of observers it just looks like HTTPS traffic.

But running SSH over port 443 makes it particularly difficult to run a secure web server as well.

So what can be done?

Well, it turns out that when an SSH client initiates a connection, it waits for the server to send a banner (SSH-1.99-OpenSSH_4.7, for example) before beginning the secure negotiation. Browsers, on the other hand, just go right ahead and begin the secure negotiation.

Knowing this, it didn't take long to formulate a plan: write a little program which will listen on 443. When it gets a connection, it figures out if the remote end is trying to speak SSH or HTTPS, makes the appropriate connection on the local end, the passes the data between the two.

It only took a couple of hours to write a small proof-of-concept... And, believe it or not, it even works! Disclaimer: this particular implementation is not quite suitable for any real use.

Anyway, this may not be the most efficient way of doing things... But that's no matter. It's still neat

Code: sstp.py

SVK, despite some of it's shortcomings, has proved worthy of my use in the last couple of days. Here is a quick script which shows how I've been using it, and how it's rocked the pants off of SVN:

red = commands with SVN
blue = equivalent SVK commands
green = comment

# Mirror the DrP repository to //dpr
$ # No equivilent
$ svk mirror https://drproject.org/drproject/DrProject/ //drp/

# Create the tags branch
$ svn cp https://drproject.org/drproject/DrProject/tunk \
         https://drproject.org/drproject/DrProject/branches/tags
$ svk cp //drp/trunk //drp/branches/tags

# Check out the branch
$ svn co https://drproject.org/drproject/DrProject/branches/tags Tags
$ svk co //drp/branches/tags Tags

# Some changes are made to the tags branch

# Checkin those changes
$ svn ci -m "Made some changes to tags branch"
$ svk ci -m "Made some changes to tags branch"
# In this case, SVK is not being used in decentralized
# mode, so the changes will be pushed upstream to the
# SVN repository, just like the svn commit does.

# Changes have also been made to trunk.
# Time to merge those changes in.

# First, a merge with SVN:
$ svn log --stop-on-copy
# Look for either the last revision which trunk was merged
# You can't always use the base revision, otherwise conflicts
# may ensue.
$ svn merge -r $REV:HEAD https://drproject.org/drproject/DrProject/tunk .
# Hope that there are no trivial conflicts to sort out,
# then test and commit the merge. 
$ svn ci -m "Merged trunk into tags."

# And now, exactly the same operation with SVK:
$ svk pull
# ... changes are merged ...
$ 

I don't think I need to say any more

(Yes, I realize that there are tools like svnmerge.py which make SVN merges less painful... But that's one extra command to run. And you've still got those stinking .svn directories everywhere.)

I'm sitting in a tutorial at PyCon¹ on hacking the Python core. One of the things that was mentioned is Python's coding standards, and their supporting Vim scripts.

I've got no complaints with them -- I just don't want to use them all the time. So I've added this bit of code to ~/.vimrc:

let path = expand('%:p:h')
if path =~ "Python-trunk"
    let path = substitute(path, '\(.*/Python-trunk\).*', '\1', '')
    execute('source ' . path . '/Misc/Vim/vimrc')
endif

In short, if it looks like I'm in the Python-trunk directory, it sources their vimrc

1: Blog posts to come... I promise!

If there is one thing that I've found universally confusing about version control systems, it's how to revert back to a previous revision, then move on from there. Conventional wisdom would dictate that svn up -r $OLD with some additional flag that says pretend that we're still at HEAD would do the trick... But, alas, there exists no such flag.

So, besides svn cat -r $OLD $FILE > $FILE, what's the best way to revert a file (or entire tree) to an older revision? Well, it turns out that merge is the tool you're looking for.

The trick is that you can diff backwards as well as forwards:

[wolever@thebes] ~/test_dr/All svn diff -r 2:1
Index: stuff
===================================================================
--- stuff       (revision 2)
+++ stuff       (revision 1)
@@ -1,4 +1,3 @@
 Let us endeavor so to live that when we come to die even the undertaker will be
 sorry.
                -- Mark Twain, "Pudd'nhead Wilson's Calendar"
-Increased knowledge will help you now.  Have mate's phone bugged.
Index: docs.html

Which means that merge will work both ways as well:

[wolever@thebes] ~/test_dr/All svn merge -r 2:1 stuff
U    stuff
[wolever@thebes] ~/test_dr/All svn ci -m "reverted changes" stuff
[wolever@thebes] ~/test_dr/All svn diff -r 1:3 stuff
[wolever@thebes] ~/test_dr/All

Of course, this will also work with the myriad of other version control tools out there. In fact, if you're still using SVN, reverting old changes is probably the least of your problems... At least compared to, say, merging different branches ^_^

But that's a post for another day -- I need to stop writing about merging and actually get on with, err, doing it...

Ah, a Saturday morning that I should spend studying for my exam on Wednesday... What better thing to do than get all my important config files under version control?

Executive summary

To get all my config files under version control, I:

1. Moved all my important config files to ~/.config/
2. Put ~/.config/ under version control
3. Wrote a script to make links from ~/.config/* to the Right Place (ie: ~/.config/vim -> ~/.vim)
4. All my machines are in sync

So, the problem: I use three machines on a regular basis (my laptop, my server, my computer at work). I want to keep all my important config files in sync (ie: Bazaar and Vim). I'd also like to keep them under version control (version control is a good thing, right?) Can both happen at once? Well, with the magic of bzr (or your favourite distributed version control tool), it turns out they can

The first step was to collect everything into one place. I moved the files I'm going to care about (in this case, ~/.vim, ~/.vimrc and ~/.bazaar) into ~/.config and put that under version control: cd ~/.config; bzr init; bzr add * (it's worth noting that I had some problems with a couple of the bzr plugins which were under version control... Bzr didn't want to add them to my repository because they themselves were repositories. That problem was fixed with a quick rm -r <plugin>/.bzr though).

The next step was replacing the old config files with links to the new (versioned) ones. That was nothing a simple shell script couldn't solve, though (take a look at install_links.sh if you're interested in now it works).

Beautiful. Everything is versioned, I've just got to distribute it.

Bazaar (and just about every other distributed VCS out there) has some wonderful facilities to push/pull it's repositories. All it took was a quick bzr push sftp://wolever.net:443/home/wolever/.config/ to move it over, then sshing over, checking out the files and installing the links:

  $ ssh wolever.net
  $ cd .config
  $ bzr co .
  $ ./install_links

Finally, I put a link to the repository in a web-facing directory (http://wolever.net/~wolever/conf) so I could grab it from my work machine using bzr get http://wolever.net/~wolever/conf/ ~/.my_config (I have to use ~/.my_config because some program (I suspect XFCE) is using ~/.config).

And that's all there is to it

Situation: DrProject has a very useful timeline feature, which shows all the activity (tickets opened/closed, source control checkins, wiki edits, etc) relating to a project. To make things even better, there is also an option to export this feed as RSS (now, [this is also slightly broken][960], but that's another story).

Problem: If students are using DrProject to manage a marked project, there should be no way that students in one group can see the timeline another group. From the web interface, this is handled by the standard authentication mechanism... But there is no standard mechanism for authenticating RSS feeds. And, even if there was, students should not be giving their CDF passwords to online news reading services (both because the service might do Something Bad with it and because people should not be in the habit of giving out their credentials).

Current solution: Turn off RSS.

That's a really crappy solution.

What can be done?

Well, Andrew Louis and I were talking about it yesterday, and we were discussing a few of the different mechanisms used by similar web pages.

Flickr, for instance, assigns each user an arbitrary ID. After they register a name, there is no way to see this ID. If you manage to find a person's ID (either before they register a name or through some other means), there is nothing stopping your from taking a look at any of their feeds (take a look at the photos from my contacts!). Ok, so that's a pretty crummy solution.

Facebook is a bit more interesting. It assigns each feed a random ID. To view a feed, you need to know both the user's ID (easy to get) and the random ID (hard to get): http://utoronto.facebook.com/feeds/friends_status.php?id=1658670036&key=c16adb2zcd&format=rss20.

But this isn't quite good enough. If someone gets hold of the secret key, it's all over.

So what can we do?

Well, else can we use to tell the difference between you and me? What about the headers our newsreaders send? The IP block our request originates from? How frequently we check the feed?

Ok, so it might be tricky to implement a frequency-checking heuristic... But it would be easy to say "hhmm... Last time you checked this feed, you used Google Reader. But this time your User-agent tells me that you are curl. Something's up!"

When a user wants to read an RSS feed, they are given a URL with a unique key. The first time that feed is accessed, the User-agent string (and possibly other headers) are recorded. On subsequent requests, the values are compared and if there is a significant change (for some value of "significant") [edit] and the old key will be invalidated. The legitimate user will be asked to re-validate the next time they download the feed.[/edit]

This will not make attacks impossible, but will certainly frustrate them, because the attacker must get the headers right on the first try. [edit] If they get it wrong, they will have to wait for the legitimate user to get a new key, find that key, then try again. [/edit]

Problems:

• What if someone checks their feed from both an online service (such as Google Reader) and a newsreader on their laptop?
  They just generate two keys -- I don't see any reason that one user should not have more than one key associated with them.
• What if the headers change from less sinister actions?
  No problem -- getting a new key will be a matter of a couple of clicks.
• But doesn't that mean all the old items will be marked unread?
  Yes. Is there a way to fix that?
• Isn't there still a possibility of attack?
  Yes, it is possible. But, hopefully, it will be hard enough to thwart casual attackers.

So, any comments?

Postscript 0: What about OpenID?

It looks really neat. In theory, it would solve this problem. In practice, though, there are two issues:

• Added complexity -- we would either have to build an OpenID server into DrP or rely on an external server.
• Limited support -- at the moment, only a handful of readers have support for OpenID (Google Reader being, I believe, a notable exception)... And we don't quite have the market share to change that. At least not yet