TLS, threads, and workers

[Ben Laurie]

Roger Dingledine wrote:

> Nick and I built a plan today. We're going to switch to TLS for links.
> This has a number of related design fallouts.
> 
> Because we want to avoid half-second hiccups for the PK crypto ops
> (non-blocking for SSL refers to network, not cpu), we must do the SSL
> handshakes in a separate thread. Because we can't pass sockets between
> processes, we're going to get rid of the current fork()ing model, and move
> to pthreads. This means we'll be at the mercy of all the funny threading
> implementations out there, but hopefully that won't be so bad. Besides,
> forking sucked too.

Threading works pretty badly on FreeBSD.

> We'll still have the async model. There will be one main thread, and
> some workers. Workers only talk to the main thread (not to each other),
> and only through a single shared work queue in memory (work units are:
> function to call, data to give it, a callback to hand back along with
> the answer). We might use a socketpair along with the work queue (shared
> by all workers), to wake up the main thread while it's polling. We'll
> leave threading details until after TLS is in.
> 
> Our read/write pollarray abstraction gets a bit more exciting, because
> SSL can sometimes return "I want to read" when you try to write, and
> vice versa. Not so bad.
> 
> We need to start using full-blown certificates rather than just PEM
> encodings of public keys. Also not so bad.
> 
> TLS supports link key rotation very easily. This is good. Link key
> rotation is generally fast (eg hashing the old key, some nonces, the
> stuff that's gone over the wire), so doesn't need to use a separate
> thread. (We still need to resolve rotating the encryption key, but we
> needed to do that anyway.)
> 
> TLS has 24 bytes overhead on every 'record'. A record could be between
> 1 and 63 cells, depending on how many cells we ask it to send at once.
> It's interesting to note that if the record is many cells, then SSL won't
> admit to having any data until the entire record has arrived (the hash
> is at the end of the record). Under normal circumstances this should
> be fine. But it could turn ugly: for example, how well does it handle
> bandwidth throttling and TCP pushback? This becomes especially important
> when limited bandwidth is spread over hundreds of connections. We will
> have to throttle in terms of reading from and writing to the SSL objects;
> I'm not about to start messing with the TLS stuff on the wire (and OpenSSL
> doesn't let us, anyway).

Actually, it does. See my state_machine example.

Cheers,

Ben.

-- 
http://www.apache-ssl.org/ben.html       http://www.thebunker.net/

"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff