[tor-dev] Performance testing using chutney (was Re: Env variables for chutney)

[Cory Pruce]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 07/04/2015 06:19 PM, teor wrote:
>
> make test-network is dominated by the 25 second delay waiting for the
Tor test network to bootstrap. So it's not going to help much.
>
> I'm working on a chutney branch to measure bandwidth on "chutney
verify", but it doesn't have any command-line arguments yet (it's all
constants in the code). I'll see if I can pull it into shape today.
> https://trac.torproject.org/projects/tor/ticket/14175

Awesome I'm excited to try it out :D

>
> Even with these bandwidth measurement changes, there's something else
to think about:
>
> chutney will measure the combined throughput of 4-5 tor instances, and
4n - 5n cpuworker threads, where n is the number of cores on your
machine. But this isn't the performance you're interested in for
multithreaded crypto changes - you want to know how a single instance +
n cpuworker threads performs. (A chutney test network is far *more*
parallel than a typical tor relay.)
>
> To get an accurate benchmark, you could run one tor instance per
machine, or, at the very least, run the client on a slow machine, and
everything else on a fast machine, so that the client's multithreaded
crypto is the limiting factor. But this seems like a lot of work, and
I'm not sure how much accuracy you'll gain.
>
> As a first step, you could minimise the number of tor instances, which
might make multithreading improvements easier to measure.
> You'll find the basic-min network helpful for this:
> ./src/test/test-network.sh --flavour basic-min
>
> Then check if you're using ~100% of all cores when you push large
amounts (100MB+) of data through the network using #14175 (when it's
done!) If you're not using 100%, then you'll be able to see any
multithreaded improvements when you run the test again. If you are
seeing 100% usage already, get more cores or more machines, and re-run
the tests.
>
> Let me know how you go with this.
I too was thinking of how I would set up the test environment and this
is probably more than I could have come up with in days! I'm all ready
to check the cpu usage when you pump out the update.

> You could also modify tor to use single-hop connections, then measure single-hop bandwidth, by
making a 1-hop connection and pushing data through it. There won't be as
much client crypto as the 3 or 4-hop scenario; and you'll still have the
client and destination on the one machine, unlike the single relay
real-world scenario. But it could be closer to real-world multithreaded
performance, as you'll only be measuring 2n threads. (Ideally, you want
to measure n threads.)
>
> You must *never* use a tor binary built like this on the public tor
network, as it has no anonymity.
> To make tor use 1-hop circuits for everything, change
DEFAULT_ROUTE_LEN to 1 in or.h
> End dire warning about loss of anonymity.
>
> Of course, 1-hop circuits might hide some subtle multithreading bugs,
as there's less crypto happening overall. So please test the correctness
of your code with DEFAULT_ROUTE_LEN 3 as well.

Done. I changed it to start with 1 and I will try 3 as well. If I
shouldn't test this out on the public tor network, should I set up
everything locally?

>
> Give me 8 hours or so to work on #14175, I'll try and get it into a
usable state.
>
> Tim

Dude, thanks a bunch for you help. I'm really excited to start :D I'm
going to read through the initial design and the code to see what
functions/structures/constants/etc. need to be changed. Let me know when
you release #14175 and I'll be happy to be the test guinea pig =-)

- - Cory
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVmWdxAAoJEB7DuCIJauCgT28P/R1WliIB8P/3tCCPuSLE1oqX
vheyu1ZaPPfYOZqemo5xvScyuoPQjASmlM6EG20IQddSrLdz3MU/eUDD604X/tmY
r4/LvrWrHek+ER3kCFYvKvYCLO94pY6Sr5zha01iZHw0Vibvypp8lFQR4HOnjvGt
LD0LOHJXwotFcHmHfdElHT3VozFhHPHqrTQlgI8ziFl+i54XPb8yuqRbT0glwX/r
WztAjiFzuMybnA18IYFRmruLk2bb9ZgQOBGmonvWEi/FVdxkuiXc6b+UyC1q9IiG
4hdDVuwVvwa4iDJ8fWm1rGd81mMlIU6FuV77UOUZYg8/4EHFfFzzVynm0V1QhXzr
/UKhTk/3alqcydDZ3rmeX0wsHpI9RKgrwxSXFTuFm+iydjZRcbeXS9ftv88jykMW
RJ7qwmRfib0e1U+Uk3yXHHnb7k1Qt1kHxG5rFlFivsXSE06rVfq2visHlLJopAxJ
0bZ93d37tDN71yi8lBmko4mcmO4iKbTjAiw42h2Nj0iYvaM8FBrAz5i65QrlgBa8
LOTjEcxgFxeglIvInmnlf3muJvKrEYH9ikF2f6Wd1ggIqBzKxPxfyhC/0EPPJ9to
PYdqHH2VzwimhwXergEec6EWnTTyYE66iDHszzLV8KZNkFa+7nFq3VNK6KneIwTD
uklR2RCxwPjUgNaOwbdF
=z2cK
-----END PGP SIGNATURE-----