[tor-dev] Performance testing using chutney (was Re: Env variables for chutney)

[teor]

> On 5 Jul 2015, at 05:45 , Cory Pruce <corypruce at gmail.com> wrote:
> 
> One more thing for right now: how should I do benchmarks with chutney.
> Should I measure the averages of how long it takes to complete the make
> test-network command?

make test-network is dominated by the 25 second delay waiting for the Tor test network to bootstrap. So it's not going to help much.

I'm working on a chutney branch to measure bandwidth on "chutney verify", but it doesn't have any command-line arguments yet (it's all constants in the code). I'll see if I can pull it into shape today.
https://trac.torproject.org/projects/tor/ticket/14175

Even with these bandwidth measurement changes, there's something else to think about:

chutney will measure the combined throughput of 4-5 tor instances, and 4n - 5n cpuworker threads, where n is the number of cores on your machine. But this isn't the performance you're interested in for multithreaded crypto changes - you want to know how a single instance + n cpuworker threads performs. (A chutney test network is far *more* parallel than a typical tor relay.)

To get an accurate benchmark, you could run one tor instance per machine, or, at the very least, run the client on a slow machine, and everything else on a fast machine, so that the client's multithreaded crypto is the limiting factor. But this seems like a lot of work, and I'm not sure how much accuracy you'll gain.

As a first step, you could minimise the number of tor instances, which might make multithreading improvements easier to measure.
You'll find the basic-min network helpful for this:
./src/test/test-network.sh --flavour basic-min

Then check if you're using ~100% of all cores when you push large amounts (100MB+) of data through the network using #14175 (when it's done!) If you're not using 100%, then you'll be able to see any multithreaded improvements when you run the test again. If you are seeing 100% usage already, get more cores or more machines, and re-run the tests.

Let me know how you go with this.

You could also modify tor to use single-hop connections, then measure single-hop bandwidth, by making a 1-hop connection and pushing data through it. There won't be as much client crypto as the 3 or 4-hop scenario; and you'll still have the client and destination on the one machine, unlike the single relay real-world scenario. But it could be closer to real-world multithreaded performance, as you'll only be measuring 2n threads. (Ideally, you want to measure n threads.)

You must *never* use a tor binary built like this on the public tor network, as it has no anonymity.
To make tor use 1-hop circuits for everything, change DEFAULT_ROUTE_LEN to 1 in or.h
End dire warning about loss of anonymity.

Of course, 1-hop circuits might hide some subtle multithreading bugs, as there's less crypto happening overall. So please test the correctness of your code with DEFAULT_ROUTE_LEN 3 as well.

Give me 8 hours or so to work on #14175, I'll try and get it into a usable state.

Tim

Tim Wilson-Brown (teor)

teor2345 at gmail dot com
pgp ABFED1AC
https://gist.github.com/teor2345/d033b8ce0a99adbc89c5

teor at blah dot im
OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20150705/c2da44e6/attachment.sig>