[tor-dev] Twisted-based Tor client performance measurement tool

[meejah]

Hi Karsten,

txtorcon has examples that do some of these things, and Twisted
includes a Web client (and Web server). I wouldn't mind helping out
here and there, but I can't commit specific time for the next little
while as I have a day job, plus am pretty busy for the next couple
months with some home improvements.

If you did want to use txtorcon, I'd be very interested to hear about
(and add) any features it lacks (using the "Issues" interface on
github.com/meejah/txtorcon would be the best place to request one).

> 1) set up local Tor clients, configure them, and register for events;
> 2) run a local web server to download files from or upload files to;

A super-simple version of this is
examples/launch_tor_with_hidden_service.py (the web server then being on
a hidden service, too). Adding timing code would be easy.

> 3) periodically run one or more tests which can be:
> 3.1) an HTTP GET request over Tor to its own web server,
> 3.2) an HTTP POST request to measure upload speed,
> 3.3) a GET or POST request to a locally running hidden service,

You'd need https://github.com/ln5/twisted-socks for a SOCKS client for
this (looks like V4 only?). There are some other ones floating around
out there, too, but nothing in core Twisted (as far as I
recall). Ah, like https://twistedmatrix.com/trac/ticket/1330

> 3.4) a series of fetches of top 50 Alexa domains using Selenium/Firefox;
> 3.5) a series of requests to track stream/circ allocations for #5830;

In the "exit_scanner" branch is some more-complicated code to do
something kind-of like this (and more), and there's also
examples/circuit_failure_rates.py -- which both sound similar but not
quite the same as what you want (but probably decent examples). The
failures rates one is way simpler (passively analyizes failures rates
per-guard). I'd guess all you'd need for the selenium stuff is
event-based subprocess launching, which Twisted does have (txtorcon
uses it to launch slave Tors).

> 4) store request timestamps and Tor controller events to SQLite;
> 5) provide results via a RESTful API over its web server.

> My questions:
> - Is Twisted the right framework for this?  What are the alternatives?

If you like event-based code, it's almost certainly the best
choice. If threaded is fine (or preferred), then there are a lot more
options (and of course, you'd then want Stem as the controller
library) but I don't know them that well. Probably it would mean
combining a Python web-client library with some suitable web-server
thing (dmaybe the built-in SimpleHTTPServer? Lukas Lueg contributed an
example for doing this with txtorcon). Depending on how much work
you're doing, beware that Python doesn't play very nicely with
threads.

Also, since you're talking about a Web service, it might be worth
looking at Cyclone for that piece (which is Tornado, the friendfeed
thing, but with their custom async-stuff taken out and Twisted's put
in). Twisted's Web server will work well for simple things (e.g. the
speed tests), but if you want a "modern" AJAX-y sort of thing, Cyclone
is likely a better choice.

-- 
meejah