This tip was originally posted to the python-dev mailing list, so I can't take one scrap of credit for it.
Basically, since OpenSSH4, there has been an option to share connections -- that is, once you've opened one connection to a host, every subsequent connection is tunneled through the same channel, completely removing the overhead of authentication!
It's quite simple, just add this to ~/.ssh/config:
ControlMaster auto
ControlPath ~/.ssh/.%r@%h:%p
There can be problems if your machine crashes and that file is left lying around... So adding this to your crontab will fix that:
@reboot rm -f .ssh/controls/*
Next time: tab completion with scp!