Redis can make a fantastic lightweight messaging system for situations where the complexity of larger systems (like, for example, RabbitMQ) isn't necessary.

producer                      | consumer 1                 | consumer 2
------------------------------+----------------------------+----------------------------
                              | > SUBSCRIBE "#dogfacts"    | > SUBSCRIBE "#dogfacts"
                              | Reading messages...        | Reading messages...
                              | 1) "subscribe"             | 1) "subscribe"
                              | 2) "#dogfacts"             | 2) "#dogfacts"
                              | 3) (integer) 1             | 3) (integer) 1
                              |                            |
> PUBLISH "#dogfacts" "hello" | 1) "message"               | 1) "message"
(integer) 2                   | 2) "#dogfacts"             | 2) "#dogfacts"
>                             | 3) "hello"                 | 3) "hello"

def send_message(channel, message):
    timestamp = now()
    db.execute("""
        INSERT INTO chat_messages (timestamp, channel, message)
        VALUES (?, ?, ?)
    """, [timestamp, channel, message])
    redis.publish(channel, {
        "message": message,
        "timestamp": timestamp,
    })

def read_messages(channel):
    pubsub = redis.pubsub()
    pubsub.subscribe(channel)
    last_timestamp = 0
    query = """
        SELECT *
        FROM chat_messages
        WHERE channel = ?
        ORDER BY timestamp ASC
    """
    for message in db.execute(query):
        yield message
        last_timestamp = message["timestamp"]

    for message in pubsub.listen():
        if message["timestamp"] <= last_timestamp:
            continue
        yield message

producer                      | consumer 1                 | consumer 2
------------------------------+----------------------------+----------------------------
> LPUSH "work-queue" "job1"   |                            |
(integer) 1                   | > BRPOP "work-queue"       | > BRPOP "work-queue"
>                             | "job1"                     |
> LPUSH "work-queue" "job2"   | >                          | "job2"
(integer) 1                   | > BRPOP "work-queue"       | > BRPOP "work-queue"
>                             |                            |
> LPUSH "work-queue" "job3"   |                            | "job3"
(integer) 1                   |                            | >
> LPUSH "work-queue" "job4"   | "job4"                     | >
(integer) 1                   | >                          | >

def run_worker(queue):
    while True:
        job = redis.brpop(queue)
        run_job(job)

> LPUSH "job-queue" "1"
> LPUSH "job-queue" "2"
> LPUSH "job-queue" "3"
> BRPOPLPUSH "job-queue" "pending-jobs"
> LRANGE "job-queue" -1 0
1) "2"
2) "3"
> LRANGE "pending-jobs" -1 0
1) "1"

def nb_notify(board, value):
    timestamp = now()
    ts_value = "%s:%s" %(timestamp, value)
    redis.set(board, ts_value)
    redis.broadcast(board, ts_value)

def nb_watch(board, last_timestamp=0):
    pubsub = redis.pubsub()
    pubsub.subscribe(board)
    ts_value = redis.get(board)

    while True:
        ts_str, _, value = ts_value.partition(":")
        ts = float(ts_str)
        if ts > last_timestamp:
            return (ts, value)
        ts_value = pubsub.wait()

def progress_view(request):
    last_ts = float(request.GET.get("ts", 0))
    next_ts, value = nb_watch("some-board", last_ts)
    return JSONResponse({
        "next_update_url": request.path + "?ts=%s" %(next_ts, ),
        "value": value,
    })

var nextUrl = "https://example.com/api/progress";
function watchProgress(callback) {
    http.get(nextUrl).then(function(result) {
        nextUrl = result.next_update_url;
        callback(result.value);
        watchProgress(callback);
    });
}

watchProgress(function(value) {
    console.log("Progress:", value);
});

I logged in to my Amazon Mechanical Turk (see also: Mechnical Turk) for the first time in years the other day:

And was reminded of the best $4 I spent in 2008: a collection of terrible Turing machine related jokes.

Below are the raw and unedited results of asking workers on Mechanical Turk to help me find Turing machine jokes. Each respondant was paid $0.25 if their response was accepted, and I was pretty liberal accepting responses:

I'd like to take a moment to share a few ways I use ~/.ssh/config file to make my life happier every day.

With these options I never need to remember host names, usernames, or port numbers, and the vast majority of my SSH commands look like:

$ ssh myapp
$ ssh myclient-prod-db
$ rsync -a app-backup:backups/jan01 .

Every time I get ssh access to a server I add an entry to my config file giving the host a name that's meaningful to me (for example, "someclient-server" or "myproj-backup") and setting the default username and port:

Host someclient-dev
    Hostname 11.22.33.44
    User dev

Host someclient-prod-app
    Hostname redbunny.myclient.com
    Port 4242
    User prod

Host someclient-prod-db
    Hostname bluefish.myclient.com
    Port 4242
    User db

These host alias can be used just about everywhere a hostname is passed to SSH, including:

• SSH from the command line:

  $ ssh someclient-dev
  ...
  dev@11.22.33.44 $
  

• git, mercurial, or other version control systems:

  $ git remote add dev someclient-dev:repo/
  

• rsync:

  $ rsync -a media/ someclient-dev:media/
  

Not only does this mean I never need to remember weird hostnames or arbitrary usernames, but I can also open the file to see a list of all the machines I've ever had access to (which can be very useful when an old machines needs work done).

The bash-completion package is even .ssh/config aware, so tab completion will work as expected:

$ ssh someclient-<tab>
someclient-dev someclient-prod-app someclient-prod-db

Amazon EC2 key management is also a huge continence. Each time I get access to an Amazon EC2 instance I add the IdentityFile to the Host definition:

Host *.amazonaws.com
    User ec2-user

Host myapp
    Hostname ec2-1-2-3-4.compute-1.amazonaws.com
    IdentityFile ~/.ssh/aws-myapp.pem

As above, this will create the host alias myapp, and the identify file ~/.ssh/aws-myapp.pem will be used to connect (no more -I flag on the command line).

Finally, there are a few options that are useful to set for all hosts:

Host *
    # Instead of just printing the host key fingerprint as an opaque hex
    # string, print a pretty art. Ostensibly this is for security, but
    # mostly it's pretty:
    #     +--[ RSA 2048]----+
    #     | oE    ..        |
    #     |  ..   ...       |
    #     |   .  ooo        |
    #     |   oooooo        |
    #     |  . =+.+S+       |
    #     |   o.+o.o..      |
    #     |    o..          |
    #     +-----------------+
    VisualHostKey yes

    # Send explicit keepalive packets. This isn't often a problem, but I've
    # run into a few combinations of network and machine that will drop
    # inactive connections.
    KeepAlive yes
    ServerAliveInterval 60

    # SSH Agent Forwarding is described here:
    # http://www.unixwiz.net/techtips/ssh-agent-forwarding.html
    ForwardAgent yes

    # SSH Control Channels allow multiple SSH sessions to share one
    # connection. For example, the first time I run "ssh myapp", ssh will
    # create a new connection to the server (creating a TCP connection,
    # authenticating, etc). As long as that connection
    # is active, though, running "ssh myapp" from another terminal will
    # re-use the same TCP connection, authentication, etc, making the
    # command virtually instant.
    # Note that the ControlPersist option is important, otherwise all the
    # sessions will be disconnected when the master session closes.
    ControlPath ~/.ssh/control/master-%l-%r@%h:%p
    ControlMaster auto
    ControlPersist 60

The dangers of Python's super have been documented... but, in my humble opinion, not well enough.

A major problem with Python's super() is that it is not straight forward to figure out needs to call it, even if it doesn't seem like the method's parent class should need to.

Consider this example, where mixins are used to update a dictionary with some context (similar to, but less correct than, for example, Django's TemplateView):

class FirstMixin(object):
    def get_context(self):
        return {"first": True}

class BaseClass(FirstMixin):
    def get_context(self):
        ctx = super(BaseClass, self).get_context()
        ctx.update({"base": True})
        return ctx

class SecondMixin(object):
    def get_context(self):
        ctx = super(SecondMixin, self).get_context()
        ctx.update({"second": True})
        return ctx

class ConcreteClass(BaseClass, SecondMixin):
    pass

This looks correct... but it isn't! Because FirstMixin doesn't call super(), SecondMixin.get_context is never called:

>>> c = ConcreteClass()
>>> c.get_context()
{"base": True, "first": True} # Note that ``"second": True`` is missing!

Alternatively, image that FirstMixin.get_context() does call super():

class FirstMixin(object):
    def get_context(self):
        ctx = super(FirstMixin, self).get_context()
        ctx.update({"first": True})
        return ctx

This will also be incorrect, because now the call to super() in SecondMixin will trigger an error, because the final base class - object - does not have a get_context() method:

>>> c = ConcreteClass()
>>> c.get_context()
...
AttributeError: 'super' object has no attribute 'get_context'

What is a poor Pythonista to do?

There are three reasonably simple rules to follow when dealing with this kind of multiple inheritance:

1. Mixins should always call super().
2. The base class should not call super().
3. The base class (or one of its super classes) needs to be at the right of sub-classe's list of base classes.

Note that this will often mean introducing an otherwise unnecessary *Base class.

To correct the example above:

# Following rule (1), every mixin calls `super()`
class FirstMixin(object):
    def get_context(self):
        ctx = super(FirstMixin, self).get_context()
        ctx.update({"first": True})
        return ctx

# Following rule (2), the base class does *not* call super.
class BaseClassBase(object):
    def get_context(self):
        return {"base": True}

# Notice that, to follow rule (3), an otherwise uneccessary base class has
# been introduced to make sure that the "real" base class (the one without
# the call to super) can be at the very right of the list of base classess.
class BaseClass(FirstMixin, BaseClassBase):
    pass

# Following rule (3), the base class comes at the right end of the list
# of base classess.
class ConcreteClass(SecondMixin, BaseClass):
    pass

This will guarantee that the mixins are always called before the base class, which doesn't call super() in get_context().

Note that this will still cause problems in the even that multiple base classess are used (ie, "true" multiple inheritance)... and there isn't much which can be done about that, at least in the general case.

It is also worth noting that that in many cases the best solution is to avoid inheritance all together, opting instead for a pattern better suited to the requirements of the specific problem at hand.

For example, in the sitaution from the example above - where many different "things" (in the above example: mixins and the base class) need to contribute to the "context" dictionary - one option which might be more appropriate is an explicit set of "context providers":

class FirstContextProvider(object):
    def __call__(self):
        return {"first": True}

class BaseClass(FirstMixin):
    context_providers = [
        FirstContextProvider(),
        lambda: {"base": True},
    ]

    def get_context(self):
        ctx = {}
        for provider in self.context_providers:
            ctx.update(provider())
        return ctx

class SecondContextProvider(object):
    def __call__(self):
        return {"second": True}

class ConcreteClass(BaseClass, SecondMixin):
    context_providers = BaseClass.context_providers + [
        SecondContextProvider(),
    ]

(recall that __call__ method is used to make instances of a call callable)

Edit: I was corrected by @lambacck, who pointed out the "base class on the right" rule: https://twitter.com/lambacck/status/451528854507905024

Googling around the other day I was disappointed to find that the internet has a few incorrect examples of how atomic bank account transfers can be implemented with CouchDB... but I wasn't able to find any correct examples.

So here it is: the internet's first 100% complete and correct implementation of the classic "atomic bank balance transfer problem" in CouchDB.

First, a brief recap of the problem: how can a banking system which allows money to be transfered between accounts be designed so that there are no race conditions which might leave invalid or nonsensical balances?

There are a few parts to this problem:

First: the transaction log. Instead of storing an account's balance in a single record or document — {"account": "Dave", "balance": 100} — the account's balance is calculated by summing up all the credits and debits to that account. These credits and debits are stored in a transaction log, which might look something like this:

{"from": "Dave", "to": "Alex", "amount": 50}
{"from": "Alex", "to": "Jane", "amount": 25}

And the CouchDB map-reduce functions to calculate the balance could look something like this:

POST /transactions/balances
{
    "map": function(txn) {
        emit(txn.from, txn.amount * -1);
        emit(txn.to, txn.amount);
    },
    "reduce": function(keys, values) {
        return sum(values);
    }
}

For completeness, here is the list of balances:

GET /transactions/balances
{
    "rows": [
        {
            "key" : "Alex",
            "value" : 25
        },
        {
            "key" : "Dave",
            "value" : -50
        },
        {
            "key" : "Jane",
            "value" : 25
        }
    ],
    ...
}

But this leaves the obvious question: how are errors handled? What happens if someone tries to make a transfer larger than their balance?

With CouchDB (and similar databases) this sort of business logic and error handling must be implemented at the application level. Naively, such a function might look like this:

def transfer(from_acct, to_acct, amount):
    txn_id = db.post("transactions", {"from": from_acct, "to": to_acct, "amount": amount})
    if db.get("transactions/balances") < 0:
        db.delete("transactions/" + txn_id)
        raise InsufficientFunds()

But notice that if the application crashes between inserting the transaction and checking the updated balances the database will be left in an inconsistent state: the sender may be left with a negative balance, and the recipient with money that didn't previously exist:

// Initial balances: Alex: 25, Jane: 25
db.post("transactions", {"from": "Alex", "To": "Jane", "amount": 50}
// Current balances: Alex: -25, Jane: 75

How can this be fixed?

To make sure the system is never in an inconsistent state, two pieces of information need to be added to each transaction:

1. The time the transaction was created (to ensure that there is a strict total ordering of transactions), and
2. A status — whether or not the transaction was successful.

There will also need to be two views — one which returns an account's available balance (ie, the sum of all the "successful" transactions), and another which returns the oldest "pending" transaction:

POST /transactions/balance-available
{
    "map": function(txn) {
        if (txn.status == "successful") {
            emit(txn.from, txn.amount * -1);
            emit(txn.to, txn.amount);
        }
    },
    "reduce": function(keys, values) {
        return sum(values);
    }
}

POST /transactions/oldest-pending
{
    "map": function(txn) {
        if (txn.status == "pending") {
            emit(txn._id, txn);
        }
    },
    "reduce": function(keys, values) {
        var oldest = values[0];
        values.forEach(function(txn) {
            if (txn.timestamp < oldest) {
                oldest = txn;
            }
        });
        return oldest;
    }

}

List of transfers might now look something like this:

{"from": "Alex", "to": "Dave", "amount": 100, "timestamp": 50, "status": "successful"}
{"from": "Dave", "to": "Jane", "amount": 200, "timestamp": 60, "status": "pending"}

Next, the application will need to have a function which can resolve transactions by checking each pending transaction in order to verify that it is valid, then updating its status from "pending" to either "successful" or "rejected":

def resolve_transactions(target_timestamp):
    """ Resolves all transactions up to and including the transaction
        with timestamp ``target_timestamp``. """
    while True:
        # Get the oldest transaction which is still pending
        txn = db.get("transactions/oldest-pending")
        if txn.timestamp > target_timestamp:
            # Stop once all of the transactions up until the one we're
            # interested in have been resolved.
            break

        # Then check to see if that transaction is valid
        if db.get("transactions/available-balance", id=txn.from) >= txn.amount:
            status = "successful"
        else:
            status = "rejected"

        # Then update the status of that transaction. Note that CouchDB
        # will check the "_rev" field, only performing the update if the
        # transaction hasn't already been updated.
        txn.status = status
        couch.put(txn)

Finally, the application code for correctly performing a transfer:

def transfer(from_acct, to_acct, amount):
    timestamp = time.time()
    txn = db.post("transactions", {
        "from": from_acct,
        "to": to_acct,
        "amount": amount,
        "status": "pending",
        "timestamp": timestamp,
    })
    resolve_transactions(timestamp)
    txn = couch.get("transactions/" + txn._id)
    if txn_status == "rejected":
        raise InsufficientFunds()

A couple of notes:

• For the sake of brevity, this specific implementation assumes some amount of atomicity in CouchDB's map-reduce. Updating the code so it does not rely on that assumption is left as an exercise to the reader.
• Master/master replication or CouchDB's document sync have not been taken into consideration. Master/master replication and sync make this problem significantly more difficult.
• In a real system, using time() might result in collisions, so using something with a bit more entropy might be a good idea; maybe "%s-%s" %(time(), uuid()), or using the document's _id in the ordering. Including the time is not strictly necessary, but it helps maintain a logical if multiple requests come in at about the same time.