TLS, threads, and workers

[Ben Laurie]

Nick Mathewson wrote:

> On Thu, 2003-09-04 at 12:20, Paul Syverson wrote:
> 
>>On Thu, Sep 04, 2003 at 03:23:35AM -0400, Roger Dingledine wrote:
>>
>>>Nick and I built a plan today. We're going to switch to TLS for links.
>>>This has a number of related design fallouts.
>>
>>I hope this is a plan rather than a done decision at this point.  I am
>>concerned that this has not been adequately discussed. I should also
>>say up front that I am unqualified to discuss details given my
>>complete ignorance of coding but I think my high-level points are
>>still cogent.
> 
> 
> Here's what we're doing, and why.  We're starting to implement TLS, but
> we haven't gone threaded yet.  I don't think we should go threaded
> until/unless these issues are settled.
> 
> The reasons in favor of TLS are pretty clear: it's a well analyzed link
> protocol that gives us integrity, forward security, and mutual
> authentication, all for free.
> 
> The reason that TLS links might argue for a multithreaded design is
> that, with all the TLS implementations I know of, I don't know any good
> way to separate the SSL handshake crypto operations from the SSL
> handshake IO operations.  Thus, if we want to put the RSA operations in
> a separate execution context (i.e., process or thread) in order to avoid
> high-latency operations, we'll need to put the socket operations there
> too.  But you can't pass sockets back and forth across processes; hence
> threads seem smart.

Ah. You can plug crypto into OpenSSL with the engines, so if you really
want to palm RSA off to another process, then its not hard (modulo your
definition of hard). I do have to question why you'd want to do that,
though. Sounds like a case of premature optimisation to me.

> Another option is to go heavily multiprocess: stick _all_ link
> encryption operations in separate processes from the main onion routing
> process.  The link encryption processes could send unencrypted cells
> back and forth from the main OR process.  This could scale poorly, by
> requiring many processes.

Again, premature optimisation, IMO.

> Another option is to accept hiccups while handshakes are going on.  If
> we later impose some kind of proof of work requirement before beginning
> the handshake, this might not be so bad.  On the box that the ORs are
> currently running on, 1024 bit RSA decrypt takes 6 ms, 1536 bit RSA
> decrypt takes 19 ms, and 2048 bit RSA decrypt takes 36 ms.
> 
> Another option is to try to hack up OpenSSL to separate the crypto parts
> from the IO parts.  I don't have a lot of confidence here--OpenSSL
> internals can get complicated, and I don't see any hooks that do what we
> want.

See above!

> Is there a better solution to this problem?  I would also like to avoid
> going multithreaded if possible, but I'm not sure it's possible.
> 
> 
>>I am very hesitant to move to a threading model without a clear idea
>>that relevant problems can be handled. I was especially concerned
>>about portability as soon as it was brought up and was not heartened
>>to see Ben's comment that FreeBSD handles threads poorly.
> 
> 
> While I don't question Ben's experience, I must add another data point
> --- Mixminion uses a lightly multithreaded design, and seems to run fine
> on FreeBSD.  ("I've never had any complaints." :P )  Ben: can you point
> me towards the discussion where apache2 decided that FreeBSD threads
> were inadequate to their needs?  Perhaps they discuss specifically what
> doesn't work right.

I don't believe there are specifics, but the general experience is that
after not very long, Apache starts eating 100% of the CPU and not
answering queries. I've discussed this with at least two FreeBSD
developers (Robert Watson [core] and Tony Finch [not core]) and they
both agreed that FreeBSD threads suck. Bigtime.

> Another data point: stunnel seems to use threads when it can.
> 
> 
>>                                                            I'm also
>>skittish because we started using threads in a previous incarnation of
>>onion routing because it seemed to make alot of sense but had to
>>switch back because of assorted problems, portability, crypto
>>libraries we wanted to allow people to choose from weren't thread
>>safe, and so on.  I would like to hear a more about why it is thought
>>that these issues will not be big problems this time.
> 
> 
> "assorted problems" -- Can't comment.
> "portability" -- If we got pthreads and windows threads, we'd have
> almost the whole market.  Of course, we'd have to stick to the common
> subset of all such implementations, which could indeed be tricky. 
> Nonetheless, there are enough examples of portable thread wrappers
> (Apache,Python,Perl,glib,stunnel,etc.) that we one of us can probably
> reverse engineer a threading discipline 
> "Non-threadsafe libraries" -- OpenSSL (which we're using now) can
> supposedly be compiled in a thread-safe mode.

It can.

> GnuTLS is supposedly
> threadsafe.  NSS is supposed to be threadsafe, if I read their
> documentation correctly.
> 
> So as you can probably tell, I currently regard threading as the least
> unworkable solution.  But if there's an even less unworkable solution,
> I'd rather do that instead.

Threads piss me off, in general. So many easy but hard-to-track-down
ways to shoot yourself in the foot.

The argument for threads appears to boil down to "when I need to talk to
a peer for the first time, I might glitch for a few milliseconds" - err
- so?

Cheers,

Ben.

-- 
http://www.apache-ssl.org/ben.html       http://www.thebunker.net/

"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff