don at nz.net

I found myself explaining this one at Curry tonight, in the context of discussing fast broadband.

Basically, if you have a reliable stream protocol like, to take a random example, TCP, and you’re not doing anything imaginative with it, you run into the following problem:

Every byte you send might need to be resent if it gets lost along the way.  So, you buffer whatever you send up until you get an acknowledgement from the other end.  Let’s say, for argument’s sake you use a 64k buffer. We call this buffer the window, and the size is the window size.

Now, let’s say you have a looooonnnngggg path between you and your remote endpoint. Let’s say it’s 200 milliseconds, or 1/5th of a second. This is pretty reasonable for an NZ-US connection — the speed of light is not our friend.

And finally, for simplicity sake, let’s say that the actual bandwidth over that path is Very High, so serialisation delays (the time taken to put one bit after the next) are negligible.

So,  if I send 64k bytes (or 512 k bits) worth of data, it takes 200 ms before I get an acknowledgement. It doesn’t matter how fast I send my 64k; I still have to stop when I’ve sent it.  200 ms later, I get a bunch of acknowledgements back, for the whole 64k (assuming nothing got dropped), and I can now send my next 64k.

So the actual throughput, through my SuperDuperBroadband connection, is 64k bytes per 200 ms, or 2.5 Mbps.

To turn this around, if I want 2.5 Mbps at 200 ms, I need a 64k byte window; if I want 5 Mbps on a 200 ms path, I’m going to need to up the window size to 5 Mbps times 200 ms = 128 k bytes.

That window size calculation is the bandwidth delay product.

There’s ‘s the theory. Pick a big window size and go fast. Except:

1. You don’t get to pick. Even if you control your application, for downloads you can ask for a bigger window size, but you don’t necessarily get it. Probably, you’ll get the smaller of what the applications at either end asked for.
2. Standard, 1981 edition TCP has the window (buffer) size that can be communicated by the endpoints maxed out at 64k. This isn’t the end of the world; in 1992 Van Jacobson and friends rode to the rescue with RFC 1323, which allows the window size to be scaled, to pretty much anything you like. But most TCP stacks come with a default window size in the 64k-ish range, and may applications don’t change it.
3. Even if both ends of a TCP session ask for and get a large maximum window size, they don’t start with it. TCP congestion control requires that everyone start slowly (it;s called slow start), and this is done by starting with a small window size, and increasing it as the acknowledgements flow in and the sending end can get an idea about how much bandwidth there is.  So if your application uses lots of short TCP sessions rather than one long one, you’ll never reach your maximum window size and therefore never saturate your connection.

What to do? It depends what you’re trying to achieve.  For file transfers, run lots of TCP sessions side by side – can anyone say BitTorrent? Local caching helps for web traffic; move the content closer, and the bandwidth delay product is less. Use a different protocol. I have to say I’ve seen quite a few UDP-based file transfer protocols come and go, because tweaking TCP parameters at both ends is usually a darn sight easier than getting a new protocol right. (see don’s law).

What it comes down to, is that if all you’re going to use your UltraSuperDuperFast broadband connection for is downloading videos from US servers, you’re going to be disappointed. The real key to making this useful is local, or at least, locally hosted, content. Preferably located right by the fibre head-ends. It’s a parallel stream to the effort to get the fibre in the ground and get it lit, and it needs to be attended to PDQ.