TLS, threads, and workers

[Bruce Montrose]

When you switch to threads, keep in mind that if one thread is handling
signals such as SIGUSR1, etc. that other threads in the same process that
are waiting via sleep(), select(), or poll() will also be interrupted. That
is to say, that the sleep(), select(), or poll() will return prematurely.
You should look at the revents field to see what value is returned by poll()
in this case.

> -----Original Message-----
> From: owner-or-dev at freehaven.net [mailto:owner-or-dev at freehaven.net] On
> Behalf Of Roger Dingledine
> Sent: Thursday, September 04, 2003 3:24 AM
> To: or-dev at freehaven.net
> Subject: TLS, threads, and workers
> 
> Nick and I built a plan today. We're going to switch to TLS for links.
> This has a number of related design fallouts.
> 
> Because we want to avoid half-second hiccups for the PK crypto ops
> (non-blocking for SSL refers to network, not cpu), we must do the SSL
> handshakes in a separate thread. Because we can't pass sockets between
> processes, we're going to get rid of the current fork()ing model, and move
> to pthreads. This means we'll be at the mercy of all the funny threading
> implementations out there, but hopefully that won't be so bad. Besides,
> forking sucked too.
> 
> We'll still have the async model. There will be one main thread, and
> some workers. Workers only talk to the main thread (not to each other),
> and only through a single shared work queue in memory (work units are:
> function to call, data to give it, a callback to hand back along with
> the answer). We might use a socketpair along with the work queue (shared
> by all workers), to wake up the main thread while it's polling. We'll
> leave threading details until after TLS is in.
> 
> Our read/write pollarray abstraction gets a bit more exciting, because
> SSL can sometimes return "I want to read" when you try to write, and
> vice versa. Not so bad.
> 
> We need to start using full-blown certificates rather than just PEM
> encodings of public keys. Also not so bad.
> 
> TLS supports link key rotation very easily. This is good. Link key
> rotation is generally fast (eg hashing the old key, some nonces, the
> stuff that's gone over the wire), so doesn't need to use a separate
> thread. (We still need to resolve rotating the encryption key, but we
> needed to do that anyway.)
> 
> TLS has 24 bytes overhead on every 'record'. A record could be between
> 1 and 63 cells, depending on how many cells we ask it to send at once.
> It's interesting to note that if the record is many cells, then SSL won't
> admit to having any data until the entire record has arrived (the hash
> is at the end of the record). Under normal circumstances this should
> be fine. But it could turn ugly: for example, how well does it handle
> bandwidth throttling and TCP pushback? This becomes especially important
> when limited bandwidth is spread over hundreds of connections. We will
> have to throttle in terms of reading from and writing to the SSL objects;
> I'm not about to start messing with the TLS stuff on the wire (and OpenSSL
> doesn't let us, anyway). Hopefully this will all have a happy ending;
> we'll just have to see how it performs.
> 
> Our dream of moving to a trivial little crypto library is now gone:
> we are married to OpenSSL (or something equally complex).
> 
> --Roger