Code Kills

Fun Little Threading Problem

2011-08-26T20:06:40Z

A fun little threading problem I ran into today: imagine a Thermometer which can issue notifications if the temperature reaches a certain threshold:

class Thermometer(object):
    def set_temperature(self, value):
        self.temperature = value

    def wait_for_temperature(self, target):
        # ... wait for `self.temperature` to be at least `target` ...
        return self.temperature

Using only AsyncResult, how can the set_temperature and wait_for_temperature methods be implemented to guarantee that multiple threads can wait_for_temperature without missing a temperature change (only one thread will be call set_temperature).

My solution:

class Thermometer(object):
    def __init__(self):
        self.temperature_changed = AsyncResult()

    def set_temperature(self, value):
        self.temperature = value
        next_change = AsyncResult()
        self.temperature_changed.set((next_change, value))
        self.temperature_changed = next_change

    def wait_for_temperature(self, target):
        changed = self.temperature_changed
        temperature = self.temperature
        while temperature < target:
            changed, temperature = changed.get()
        return temperature

I originally encountered this problem working on a component at work, but I enjoyed the solution so I thought I would adapt it and post it here.

Making Settlers of Catan less painful with pity points

2011-02-27T23:02:00Z

If you've played Settlers, you know that missing a couple of early-game rolls can be the difference between a win and a painful, boring loss. Pity points[0] help fix that.

The rules are simple:

Any time a player doesn't get any resources from a normal (ie, start-of-the-turn) dice roll, they get one pity point.
At any time, a player may exchange pity points equal to their number of victory points for one resource.

For example: on Alice's turn, she rolls a 3. Neither Alice nor Bob have a settlement adjacent to a 3, so they both get one pity point. Also on Alice's turn, Bob trades three pity points (because Bob has three victory points) for one sheep, and trades that sheep to Alice.

[0]: introduced to me by Fraser Harris — thanks!

Lightroom: Export a List of Pictures in a Catalog

2010-09-12T16:52:00Z

I recently had to export all the names of images in a Lightroom catalog. There was no obvious way of doing this through the Lightroom interface, so I threw together a little Python script that directly reads Lightroom's SQLite database. There are a few caveats, noted in the file (below)… But it worked pretty well for me, and hopefully other people will find it useful too.

How to Beat SimTower

2009-03-15T20:32:00Z

SimTower -- we've all played it, many of us have loved it, none of us have beat it.

Until now.

So, here we go: the definitive guide to beating Sim Tower!

0: Get the Game

It's hard to beat a game you can't play, so getting a copy of the game is a good place to start. I got mine at Abandonia.

A note here: the game will run under Darwine on Intel-based Macs... Just don't try and save -- that doesn't work.

1: Starting your Tower: hitting two stars

After the game is installed and started, begin your tower by using the build-a-lobby-in-the-bottom-left-corner-of-the-screen trick to double your money, then build a lobby that is the entire width of the screen.

After you have built the mega-lobby, start building offices. Lots of offices. Three floors worth of offices. In addition to offices, build stairs. I build them approximately half a screens width from the right side of the screen, then every screens width after that. It doesn't really matter, though -- all that matters is that you've got three floors of solid offices and stairs going to all of them.

By the time this is done, you should have two stars (300 population).

2: Fast food: three stars

Great -- you've got two stars. Only three more to go. And how will we get there? Fast food.

Why fast food? Well, the hardest part of the star requirements is the population, and fast food restaurants have the highest population to space ratio. Also, no one seems to notice if the only thing in your tower is fast food and office space.

So, time to start building!

Your first order of business is to build two security offices. It doesn't really matter where you put them, all that matters is that you have them -- they are the only requirement for three stars. Stick them under ground, at the beginning of a floor -- where ever.

Now that that's out of the way, it's time to start building fast food.

Your goal is to build eleven (yes, 11) floors of fast food.

To do this, you will also need elevators. Lots of elevators. You will need about six, distributed across the length of your building. I like to build them in pairs:

Additionally, you will need to make sure that the lazy sims working in the offices don't use these elevators. Use the inspector tool to look at each elevator, then click the floor numbers to disable floors 2,3,4. Also, I've found it helpful to have one elevator which goes to odd-numbered floors and another which goes to even-numbered floors. Try that too.

Oh, yea, and make sure that all the elevators have eight cars in them.

At some point in the process of building this unholy pile of fast food, you'll get a population of 1000 and hit three stars. Cool! On to the next step.

3: Cinemas: A (brief) change from fast food (four stars)

Once you've hit three stars you'll get the option to build even more exciting things like restaurants, shops and party halls.

But just go ahead and ignore those.

The only things you should be concerned with are:

Building more fast food.
Add a liberal sprinkling of cinemas - I have no evidence for this, but I think they bring more people into your building.
Build a few parking spots. Make sure you build enough that there are always a few empty. You will need them for the VIP and five stars.
Build one hotel suite, along with housekeeping right beside an elevator. This is also for the VIP.
Your sims will whine about wanting medical and recycling centers. Give them one medical center, and as many recycling centers as they need to be happy (2 or 3? Just don't over-do it -- recycling centers have a steep annual fee).

Alright, and once you've done that, just go back to building more fast food.

At some point here, you will hit a population of 5,000, the VIP will come and you will get four stars. Good job.

And if you get to 14 floors and still don't have four stars, just skip down to the next section anyway.

Oh, your building will catch on fire too. Just pay to have it put out.

4: Metro station, express elevators and more fast food (five stars)

Ah, four stars. You think you're almost there, don't you? Ha!

By now, you should be up to about 14 floors. You will finally get a break from the tedium of building fast food as you get to build a sky lobby and revamp your elevator system.

So here's what to do:

Build a sky lobby on the 15th floor.
Build express elevators from the 1st floor to the 15th. I have six, also in pairs.
Extend two of the express elevators all the way down, as low as they will go, then build a metro station there.
Move the existing elevators down so they go only to the 10th floor.
Build escalators from the sky lobby down to the 11th floor.
Build four more floors of fast food (from floor 16 to 20), and use escalators to connect all of those to the lobby.
Build more elevators on the 15th floor, directly over top of the elevators which go from floor 1 to 10. These elevators will be used to get people to floor 21 and above, so make sure they don't stop at floors 16 through 20).
As you build more floors of fast food, use these new elevators to connect them to the sky lobby.

And there you have it -- do this up to floor 23 or 24, and eventually you'll hit a population of 10,000 and five stars. Huzza!

5: One last thing...

Phew, you're at five stars and you've probably got a population of close to 15,000. Your first order of business is to get your population up and over 15,000 (the requirement for a "Tower" ranking). Keep building fast food until you've got that

Once you're over 15,000 population, you've only got to do one more thing: build a cathedral. To do this, pick two express elevators which are at least the width of a cathedral apart, then extend them up all the way to the 100th floor. Build a small lobby on the 90th floor, place the cathedral on the 100th floor, then build an elevator to connect them.

Now you've just got to wait for someone to get married, and your tower will be awarded the rank of "Tower"! (just don't forget to turn off fast mode so you can enjoy the wedding).

And that's it! You're done

Resources

A couple of useful resources:

Python Brain-Teaser

2008-05-29T15:14:00Z

I'm working on pulling some functionality out of one object and putting it in another, and I came across this interesting problem:

class Foo:

me = "foo"

class Bar:

me = "bar"

def get_me(self):

return self.me

Foo.get_me = Bar.get_me

x = Foo()

print x.get_me()

What does this print?

And, next question, why is that?

After lunch I'll post my thoughts

Until now, I had assumed that the semi-magic self variable was set on method calls (ie, when x.get_me() is called)... But apparently it's set when the object is instantiated (which makes perfect sense, otherwise getattr(Bar(), 'get_me')() would not work).

So I can only presume that something equivalent to this happens when an object is instantiated:

from functools import partial

new_obj = Class()

for (key, val) in new_obj.__dict__.items():

if not callable(val): continue

setattr(new_obj, key, partial(val, new_obj))

SSH Connection Sharing

2008-05-17T13:35:00Z

This tip was originally posted to the python-dev mailing list, so I can't take one scrap of credit for it.

Basically, since OpenSSH4, there has been an option to share connections -- that is, once you've opened one connection to a host, every subsequent connection is tunneled through the same channel, completely removing the overhead of authentication!

It's quite simple, just add this to ~/.ssh/config:

ControlMaster auto
ControlPath ~/.ssh/.%r@%h:%p

There can be problems if your machine crashes and that file is left lying around... So adding this to your crontab will fix that:

@reboot rm -f .ssh/controls/*

Next time: tab completion with scp!

SSH and HTTPS on the same port?!

2008-04-11T21:38:00Z

A friend of mine was over the other night, and we were talking about how much we like running SSH on port 443. It gets you around all but the most insane firewalls, and to anyone but the keenest of observers it just looks like HTTPS traffic.

But running SSH over port 443 makes it particularly difficult to run a secure web server as well.

So what can be done?

Well, it turns out that when an SSH client initiates a connection, it waits for the server to send a banner (SSH-1.99-OpenSSH_4.7, for example) before beginning the secure negotiation. Browsers, on the other hand, just go right ahead and begin the secure negotiation.

Knowing this, it didn't take long to formulate a plan: write a little program which will listen on 443. When it gets a connection, it figures out if the remote end is trying to speak SSH or HTTPS, makes the appropriate connection on the local end, the passes the data between the two.

It only took a couple of hours to write a small proof-of-concept... And, believe it or not, it even works! Disclaimer: this particular implementation is not quite suitable for any real use.

Anyway, this may not be the most efficient way of doing things... But that's no matter. It's still neat

Code: sstp.py

Problem E - Enhancing IPSC Rules: Rock Paper Scissors with PNGs

2007-07-22T03:33:00Z

Rock paper scissors in PNG form

At this point, it seems like the Facebook problems have started to go down hill. Even when Ted and I combine our attention spans, we just can't quite make it through the descriptions. To make matters worse, the only short problem left, Easy Puzzle Mountain, isn't actually all that easy...

So what ever will we do on a lovely Saturday night? Well, why not revisit an older problem. This time, it is courtesy of the IPSC¹: Problem E - Enhancing IPSC Rules.

The problem is simple: the input is a JPEG with many pairs of rocks, papers and scissors. Each pair represents one round of Rock Paper Scissors. You must figure out who won each round (and ultimately the game).

As normal for the IPSC, it was possible to solve the simple data set by hand -- each image had about 10 rounds each. The hard data set, on the other hand... Well, you can see for yourself.

10% of one hard dataset. There are 22 sets in total.

We quickly decided that it would be silly to try and examine the shape of each object, so we decided to focus on the color.

Looking at the color, it's easy to tell the difference between the rocks and everything else: they are really dark, everything else is really bright. We called this the luminosity², and it is simple to obtain: simply add the red, green and blue channels of each pixel³.

The next problem we had was telling the difference between paper and scissors. Their "luminosity" is similar, and probably not enough to make a sure match. Fortunately, the paper was fairly bland and the scissors were fairly colorful. We came up with a metric called "color delta" to measure this. The formula we used simple: color_delta = abs(red - green) + abs(red - blue) + abs(blue - green) (where abs(x) is the absolute value of x).

Alright, great. So now we have a method of distinguishing between rocks, papers and scissors... But how do I read the file? I have never worked with graphics before. The Python graphics libraries seemed intimidating, and I'll be darned if I'm also going to decode the PNGs.

Fortunately, I had the presence of mind to remember the PNM format. The Wikipedia page has more information, but all that's important for now is that the P stands for Portable, and this portability is due to the fact that it uses flat ASCII to describe an image. Nothing could be easier to parse.

for FILE in *.png
do
convert $FILE{,.pnm}
done

After this, the code came surprisingly quickly... And, short of an incredibly stupid bug (I used something along the lines of side = count / width % 2 to figure out which side of the image was being processed), it even worked! I was able to plot the output (which was the luminosity, color delta pairs) in gnuplot and see a strong correlation.

To figure out what each point was, I simply opened up one of the files in Photoshop and cut out all the items but the scissors, then all the items but papers. After adding these to the plot, it was simple to tell which group was which.

The final plot. Red points are data points, green points are papers from the training set and blue points are scissors from the training set.

And that is it

Code and sample data can be found at /code/e/.

1: On a side note, the IPSC is a great programming/problem solving contest. I have written it for the past three of four years, and every time it's a lot of fun.

2: Yes, I realize that this is completely incorrect. Brightness would be much more appropriate.

3: In actuality, I ended up using (255*3)-(r+g+b) so that white (empty) pixels would have a value of 0.

Solving Gridflip

2007-07-05T21:45:00Z

Ted and I had so much fun solving the last Facebook problem that we decided to do another this weekend: gridflip.

The problem is simple: you are given a grid of numbers, some positive and some negative. The goal is to minimize the sum of the grid, while keeping the sums of the individual rows and columns positive. The only operation that can be performed is flipping the sign of the individual rows of columns. For instance, given the grid:

 [  2  -3 ] -1
 [ -3   5 ]  2
   -1   2    2

A flip to the first row and the first column would result in the grid:

 [ -2   3 ]  1
 [ -3   5 ]  2 # After flipping the first row
   -5   8    6

 [  2   3 ]  5
 [  3   5 ]  8 # After flipping the first column
    5   8   26

This is the best solution for this grid, because it is the only solution where the sums of all the rows and columns are positive.

The solution we came up with for this problem was almost as simple: for every possible combination of flipped/not flipped columns¹, make sure that every row is positive. If, after making the sum of every row positive, the sum of every column is positive, the grid is a contender for the smallest sum.

Now, there is one small problem: rows which have a sum of zero. If a row's sum is zero, a flip won't change the value of the row, but it will change the value of the columns. The obvious solution is simply to keep track of which rows have a sum of zero, then try flipping them... And that's exactly what we did.

Now, the biggest downside to this algorithm is that it runs in O(2ⁿ) where n is the shortest side of the grid... But in practice, even my less-than-optimal Python code could solve the "large" data set in less than three seconds (Ted's C# code was almost instant).

It makes me wonder, though, is there any way to speed this up?

See /code/facebook/gridflip/ for source code and test grids.

1: Columns for simplicity -- the actual code makes sure there are fewer columns than rows by transposing the grid.

Edit: So, it appears that I can't add. I've fixed up the example grids so that they actually make sense.

Edit by Ted: I'd like to try to clarify our algorithm a little since the explanation is crappy. Let's say we have a 10 x 20 grid. First we chose the smallest dimension which in this example is 10. Then we enumerate all possible ways of the 10 rows which is 2^10 = 1024 ways. For each of those 1024 ways, we look at the 20 columns and update their sums. If any sum is < 0 we flip the column and now its sum is positive. This allows us to directly solve the pattern the columns need to be flipped in that can give a valid solution.
The only issue though is when the sum is zero. In this case we don't know whether to flip or not. Maybe out of the 20 columns, 3 will sum to zero (very few sum to zero and it does vary based on permutations but stays quite low) so that is 2^3 which is 8 possibilities. In the end the sign of the sums is checked and it is determined if we are the max sum. Only a few thousand possibilities and the operations during the iteration are quite simple (adding a few numbers and so if-else's). This results in a very fast runtime.

Sorting out Sorting

2007-06-20T14:34:00Z

Back in grade 10, my high school computer science teacher showed us a video called Sorting out Sorting. Produced at U of T in 1981¹, it shows how different sorting algorithms work and compares their runtime.

It was very exciting to see it again this morning! Check out Sorting out Sorting on YouTube.

1: http://www.cs.princeton.edu/~ah/alg_anim/animation/bibliography3_8.html#SOS