TLS, threads, and workers

[Roger Dingledine]

On Thu, Sep 04, 2003 at 04:55:47PM -0400, Nick Mathewson wrote:
> The reasons in favor of TLS are pretty clear: it's a well analyzed link
> protocol that gives us integrity, forward security, and mutual
> authentication, all for free.

I think it's clear that we want the features TLS gives us for our link
encryption. And if we continue to roll our own link handshaking we're
never going to be able to call this a mature design. So as far as I can
tell it's move to TLS now or move to TLS later; and later is usually
harder for these things.

We'd been ignoring our archaic handshake in favor of other issues;
Joel and Ben have spurred us to address it. :)

> The reason that TLS links might argue for a multithreaded design is
> that, with all the TLS implementations I know of, I don't know any good
> way to separate the SSL handshake crypto operations from the SSL
> handshake IO operations.  Thus, if we want to put the RSA operations in
> a separate execution context (i.e., process or thread) in order to avoid
> high-latency operations, we'll need to put the socket operations there
> too.

I don't know about this either way. Ben, your statemachine ssl demo seems
to be able to yield the encrypted bytes, and then I can put them on the
network myself. Is that correct? In other words, is there a way for me
to treat the SSL object as an encryptor/decryptor, and keep it separate
from the fd?

(But even if this is possible, we'd have to have some way to squeeze
an SSL object through a pipe and then back again along with the
answer. Yuck.)

>  But you can't pass sockets back and forth across processes; hence
> threads seem smart.
> 
> Another option is to go heavily multiprocess: stick _all_ link
> encryption operations in separate processes from the main onion routing
> process.  The link encryption processes could send unencrypted cells
> back and forth from the main OR process.  This could scale poorly, by
> requiring many processes.

Yep. My current thoughts are that we should hope to get an onion
routing network of 200-300 routers. After that we will need to solve the
restricted topology problem, because we can't expect to stay a clique
past that. So with the current code I'm looking to handle maybe 500
connections. Separate processes for each would just be nuts.

> Another option is to accept hiccups while handshakes are going on.  If
> we later impose some kind of proof of work requirement before beginning
> the handshake, this might not be so bad.  On the box that the ORs are
> currently running on, 1024 bit RSA decrypt takes 6 ms, 1536 bit RSA
> decrypt takes 19 ms, and 2048 bit RSA decrypt takes 36 ms.

This is a fine mid-term solution (after all, we accept the hiccups right
now). There's no rush to add threading.

We should also recognize that we're already partway down this path: we
are already multiprocess, and we have infrastructure (albeit socketpairs,
not memory queues) for communicating to/from the workers. On Win32 we
already use threads (Win32 doesn't support multiprocess well at all),
though again these are Windows threads rather than pthreads. But overall
I think this may be a smaller bullet to bite than it sounds.

> Another option is to try to hack up OpenSSL to separate the crypto parts
> from the IO parts.  I don't have a lot of confidence here--OpenSSL
> internals can get complicated, and I don't see any hooks that do what we
> want.

Assuming it's not already done (Ben?), if we do this we will want to
do it such that we can wrap the changes back into the official OpenSSL
tree. Otherwise OpenSSL will change out from under us and we'll be
constantly fighting to keep synced. We would need another developer for
this (and if we get another developer, we have better things to work on).

> While I don't question Ben's experience, I must add another data point
> --- Mixminion uses a lightly multithreaded design, and seems to run fine
> on FreeBSD.

Mixminion uses python2's thread layer. Does it use the native pthreads
directly on FreeBSD?

> "Non-threadsafe libraries" -- OpenSSL (which we're using now) can
> supposedly be compiled in a thread-safe mode.

Wait, what do you mean 'can be compiled'? :) Remember that we're using
OpenSSL because everybody already has it.

> So as you can probably tell, I currently regard threading as the least
> unworkable solution.  But if there's an even less unworkable solution,
> I'd rather do that instead.

Robust software engineering is hard.

--Roger