[metrics-bugs] #34231 [Metrics/Onionperf]: Document and maybe improve how we're mapping TGen transfers to Tor streams/circuits

[Tor Bug Tracker & Wiki]

#34231: Document and maybe improve how we're mapping TGen transfers to Tor
streams/circuits
-------------------------------+--------------------------------
 Reporter:  karsten            |          Owner:  metrics-team
     Type:  enhancement        |         Status:  new
 Priority:  Medium             |      Milestone:
Component:  Metrics/Onionperf  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:  #33328             |         Points:
 Reviewer:                     |        Sponsor:  Sponsor59-must
-------------------------------+--------------------------------

Comment (by acute):

 At the moment, Onionperf uses `stem` to log events from the Tor control
 socket corresponding to Onionperf's tor process, and later parses these
 logs (we refer to them as `torctl` logs) line by line at analysis time
 into `CircuitEvents`, `StreamEvents`, `BandwidthEvents` and
 `BuildTimeoutSetEvents`.

 The `StreamEvent` is used to extract the port which originated the
 connection (source port) and circuit ID, which is what we currently use
 for matching. There don't seem to be any other useful `StreamEvent`
 variables that can help with matching (see
 https://stem.torproject.org/api/response.html).

 However, I believe we can match `tgen` streams to Tor circuits in the
 `torctl` logs directly using SOCKS authentication.

 `Tgen 1.0.0` supports generating random usernames and passwords for SOCKS
 authentication, which can be be used to uniquely identify a transfer and
 match it to a `CircuitEvent` (`stem` already fills the `socks_username`
 and `socks_password` fields during parsing anyway).

 I've done a quick test to check,  this is how the log lines look like if
 we enable the random SOCKS authentication strings in `tgen`:

 {{{
 2020-05-23 18:01:14 1590253274.675001 [info] [tgen-transport.c:771]
 [_tgentransport_receiveSocksAuth] socks server localhost:127.0.0.1:34810
 authentication succeeded with username='zRhBJ8o' and password='zRhBJ8o'
 }}}

 ...and this is a sample line from the corresponding `torctl` log:
 {{{
 2020-05-23 18:01:17 1590253277.57 650 CIRC 406 EXTENDED
 $87C08DDFD32C62F3C56D371F9774D27BFDBB807B~Unnamed,$B9E7A637B00BBB77853A639CC33245A2FEB8F033~theykilledaaron,$3E13E2EB87CCF5690564EE33E9F9F9F80B229FBB~hotzenplotz
 BUILD_FLAGS=IS_INTERNAL,NEED_CAPACITY PURPOSE=HS_CLIENT_REND
 HS_STATE=HSCR_CONNECTING
 REND_QUERY=afa4fswz3ifwlbwsgk6va7vbbxj35m3geo3hvpc5u22w66yadr6xfayd
 TIME_CREATED=2020-05-23T17:01:16.357678 SOCKS_USERNAME="zRhBJ8o"
 SOCKS_PASSWORD="zRhBJ8o"
 }}}

 As far as the code goes, the change to the Onionperf parsers seems simple,
 and this is a better way of matching.

 Some questions/thoughts:

   * Turning on SOCKS authentication in Onionperf means we use stream
 isolation. My understanding is that each transfer (stream) would use a
 different circuit, which is what we expect anyway in Onionperf? Would this
 change affect measurements?

   * Is it likely that the `tgen` generated SOCKS credentials would
 conflict?

   * If we have plans to change what we use to parse Onionperf logs, we
 should check the replacements support this.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/34231#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online