[tor-dev] Twisted-based Tor client performance measurement tool

[Karsten Loesing]

On 1/22/13 8:32 AM, meejah wrote:
> 
> Hi Karsten,

Hi meejah,

> txtorcon has examples that do some of these things, and Twisted
> includes a Web client (and Web server). I wouldn't mind helping out
> here and there, but I can't commit specific time for the next little
> while as I have a day job, plus am pretty busy for the next couple
> months with some home improvements.
> 
> If you did want to use txtorcon, I'd be very interested to hear about
> (and add) any features it lacks (using the "Issues" interface on
> github.com/meejah/txtorcon would be the best place to request one).

Sounds great!  I avoided talking to Tor's control port, but I'll sure
have a look at txtorcon when doing that.

>> 1) set up local Tor clients, configure them, and register for events;
>> 2) run a local web server to download files from or upload files to;
> 
> A super-simple version of this is
> examples/launch_tor_with_hidden_service.py (the web server then being on
> a hidden service, too). Adding timing code would be easy.

Neat, somebody already did this. :)  Hidden services are pretty low on
my priority list, but I'll give that a look once it's at the top.

>> 3) periodically run one or more tests which can be:
>> 3.1) an HTTP GET request over Tor to its own web server,
>> 3.2) an HTTP POST request to measure upload speed,
>> 3.3) a GET or POST request to a locally running hidden service,
> 
> You'd need https://github.com/ln5/twisted-socks for a SOCKS client for
> this (looks like V4 only?). There are some other ones floating around
> out there, too, but nothing in core Twisted (as far as I
> recall). Ah, like https://twistedmatrix.com/trac/ticket/1330

I used Linus' twisted-socks, and after a tiny tweak it works quite
nicely.  The next step would be add more timestamps to it, but I first
wanted to take a closer look at the code that Arturo suggested later on.

>> 3.4) a series of fetches of top 50 Alexa domains using Selenium/Firefox;
>> 3.5) a series of requests to track stream/circ allocations for #5830;
> 
> In the "exit_scanner" branch is some more-complicated code to do
> something kind-of like this (and more), and there's also
> examples/circuit_failure_rates.py -- which both sound similar but not
> quite the same as what you want (but probably decent examples). The
> failures rates one is way simpler (passively analyizes failures rates
> per-guard). I'd guess all you'd need for the selenium stuff is
> event-based subprocess launching, which Twisted does have (txtorcon
> uses it to launch slave Tors).

Thanks for the pointers!

>> 4) store request timestamps and Tor controller events to SQLite;
>> 5) provide results via a RESTful API over its web server.
> 
>> My questions:
>> - Is Twisted the right framework for this?  What are the alternatives?
> 
> If you like event-based code, it's almost certainly the best
> choice. If threaded is fine (or preferred), then there are a lot more
> options (and of course, you'd then want Stem as the controller
> library) but I don't know them that well. Probably it would mean
> combining a Python web-client library with some suitable web-server
> thing (dmaybe the built-in SimpleHTTPServer? Lukas Lueg contributed an
> example for doing this with txtorcon). Depending on how much work
> you're doing, beware that Python doesn't play very nicely with
> threads.

I don't mind much about event-based vs. threaded.  Concurrency is always
hard to get right.  But it seems Twisted comes with a lot of
infrastructure out of the box, which may lead to fewer lines of code.

> Also, since you're talking about a Web service, it might be worth
> looking at Cyclone for that piece (which is Tornado, the friendfeed
> thing, but with their custom async-stuff taken out and Twisted's put
> in). Twisted's Web server will work well for simple things (e.g. the
> speed tests), but if you want a "modern" AJAX-y sort of thing, Cyclone
> is likely a better choice.

Cyclone might be overkill, and last I checked it wasn't in Debian's
stable repositories.  If the fancy RESTful API for results is the only
reason for using Cyclone, I think I'd rather write a simpler results
interface.

Thanks for your suggestions!

Best,
Karsten