[tor-dev] Proposal 316: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)

[Matt Traudt]

Thanks for the review, Teor. We really appreciate it.

Comments/responses inline with some trimming at the beginning (I gave up
and just left everything in after the first couple of responses).

On 4/23/20 21:05, teor wrote:
> ...
> 
> But I'm particularly concerned about any communication between
> bandwidth coordinators. Our general principle is that directory
> authorities should be independent, and we should minimise their
> communication and dependencies. This principle also extends to
> bandwidth authorities.
> 
> For FlashFlow, here are some specific reasons to avoid bandwidth
> coordinator communication:
> * it adds complexity to the protocol
> * it adds an additional failure mode: failure of coordinator
>   communication
>   * if the communication is required, this failure mode becomes a
>     denial of service vulnerability
>   * if the communication is optional, the failure could activate a
>     less-tested fallback mode, and change coordinator behaviour
> * it adds a class of additional bugs: coordinator miscommunication,
>   including race conditions
> * it adds a class of additional security vulnerabilities, via
>   coordinator communication
> * it adds additional coordinator configuration, which must stay
>   synchronised. There's two ways to sync config:
>   * in the consensus: the coordinator IP addresses are public, or
>   * privately: the configs easily get out of sync
> 

We do not envision inter-coordinator communication other than consensus
parameter voting and rare out-of-band human-to-human "hey we should
change X parameter because [...]".

Each coordinator can calculate every coordinator's measurement schedule
for the entire measurement period independently given only inputs
present in the consensus (e.g. the shared random value). I believe the
under-specified section *5.2 Measurement Scheduling* is the primary
source of your concern here.

>> On 24 Apr 2020, at 04:48, Matt Traudt <pastly at torproject.org> wrote:
>>
>> ...
>>
>> The measure commands are:
>>
>>  0 -- MSM_PARAMS    [forward]
>>  1 -- MSM_PARAMS_OK [backward]
>>  2 -- MSM_ECHO      [forward and backward]
>>  3 -- MSM_BG        [backward]
>>  4 -- MSM_ERR       [forward and backward]
> 
> Readability note:
> 
> "MSM" is a standard abbreviation for "mainstream media".
> 
> A standard abbreviation for measurement is "MEAS":
> https://www.abbreviations.com/abbreviation/MEASurement
> 

Fair, why-didn't-I-think-of-that point.

>> ...
>>
>> 3.1.3 Pre-Measurement Handshaking/Starting a Measurement
>>
>> The coordinator connects to the target relay and sends it a MSM_PARAMS
>> cell.
> 
> How much of the tor link protocol does the coordinator implement?
> 
> Currently, tor requires the following cells:
> * VERSIONS
> * NETINFO
> 
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n542
> 

Coordinators use a Tor client (with requisite new features) to
communicate with the relay. So, none of the link protocol.

>> If the target is unwilling to be measured at this time or if the
>> coordinator didn't use a TLS certificate that the target trusts, it
>> responds with an error cell and closes the connection. Otherwise it
>> checks that the parameters of the measurement are acceptable (e.g. the
>> version is acceptable, the duration isn't too long, etc.). If the
>> target is happy, it sends a MSM_PARAMS_OK, otherwise it sends a MSM_ERR
>> and closes the connection.
>>
>> Upon learning the IP addresses of the measurers from the coordinator in
>> the MSM_PARAMS cell, the target whitelists their IPs in its DoS
>> detection subsystem until the measurement ends (successfully or
>> otherwise), at which point the whitelist is cleared.
>>
>> Upon receiving a MSM_PARAMS_OK from the target, the coordinator will
>> instruct the measurers to open their TCP connections with the target. If
>> the coordinator or any measurer receives a MSM_ERR, it reports the error
>> to the coordinator and considers the measurement a failure. It is also a
>> failure if any measurer is unable to open at least half of its TCP
>> connections with the target.
>>
>> The payload of MSM_PARAMS cells [XXX more may need to be added]:
>>
>>  - version       [1 byte]
> 
> What are the minimum and maximum valid values for this field?
> 0..255 ?
> 1..255 ?
> 
> Tor uses a standard ext-type-length-value format for new cell
> fields, rather than parsing them based on a version field.
> 

You've provided lots of feedback consisting of

- min/max value questions
- suggestion of ext-type-length-value format

And it's all appreciated and valuable. We're not as up to speed on the
latest Tor coding conventions.

> It may still be useful to have a version field for information
> purposes. (And to workaround bugs in older versions.) Normally, we'd use
> Tor Relay protocol versions, but the coordinators and measurers aren't in
> the consensus.
> 
> Here's an example of the ext-type-length-value format:
> 
>      N_EXTENSIONS     [1 byte]
>      N_EXTENSIONS times:
>         EXT_FIELD_TYPE [1 byte]
>         EXT_FIELD_LEN  [1 byte]
>         EXT_FIELD      [EXT_FIELD_LEN bytes]
> 
> https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n1518
> 
> You'd want an extension in each measurer info, and also one at the end
> of the cell, for any new general fields.
> 
> Tor already knows how to parse these fields, because they are used for
> v3 onion services.
> 
>>  - msm_duration  [1 byte]
> 
> What are the minimum and maximum valid values for this field?
> 1..255 ?
> 
> Do we want to limit measurements to 4 minutes at a protocol level?
> 
> In general, protocols should make invalid states impossible to represent.
> But do we want a 4 minute hard limit here?
>

This document suggests a measurement duration of 30 seconds. We see no
reason to ever go above 1 minute. If there's a byte to spare, then sure
let's make this a uint16.

>>  - num_measurers [1 byte]
> 
> What are the minimum and maximum valid values for this field?
> 1..255 ?
> 1..17 ?
> 
> If you're using a standard 509 byte payload, then the practical limits
> are:
>   * 84 in the original format
>   * 50 with link specifiers, extensions, and IPv4 addresses
>   * 17 with link specifiers, extensions, IPv4, and IPv6 addresses
> 
>>  - measurer_info [num_measurers times]
>>    - ipv4_addr   [4 bytes]
> 
> What about IPv6 ?
> * 30% of tor relays support IPv6
> * proposal 311 introduces IPv6 connections between relays, and we're
>   implementing it right now
> * IPv4 and IPv6 routing can be different, so their bandwidths
>   can also be different
> 
> Instead of a hard-coded IPv4 field, you could use the IPv4 and
> IPv6 link specifiers, which tor already knows how to parse:
> 
>        NSPEC      (Number of link specifiers)     [1 byte]
>          NSPEC times:
>            LSTYPE (Link specifier type)           [1 byte]
>            LSLEN  (Link specifier length)         [1 byte]
>            LSPEC  (Link specifier)                [LSLEN bytes]
> 
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n1001
> 
> As an example, the v3 onion service spec re-uses link specifiers here:
> https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n227
> 

This is useful, thanks. Yes this is probably a better idea. A maximum of
17 measurers should be enough. It's hard to imagine a bwauth would have
more than even 10 hosts they'd want to act as measurers.

>>    - num_conns   [2 bytes]
> 
> This field should probably go before the link specifiers.
> 
> What are the minimum and maximum valid values for this field?
> 1..65535 ?
> 
> Do we really need to allow more than 255 connections?
> Most relays can only handle around 10,000 - 20,000 connections.
> 

160 connections was determined "to be right" (major glossing over
details here) in the experiments we did for the paper. As part of that
we also tried 320, which is bigger than 255. While I don't imagine we'll
ever want more than 255, two bytes leaves us room.

>> version dictates how this MSM_PARAMS cell shall be parsed. msm_duration
>> is the duration, in seconds, that the actual measurement will last.
>> num_measurers is how many measurer_info structs follow. For each
>> measurer, the ipv4_addr it will use when connecting to the target is
>> provided, as is num_conns, the number of TCP connections that measurer
>> will open with the target. Future versions of FlashFlow and MSM_PARAMS
>> will use TLS certificates instead of IP addresses.
> 
> FlashFlow won't be able to measure relays behind a NAT, if it
> authenticates using IP addresses. Relays see the IP address of the NAT
> device, rather than the IP address of the remote measurer.
> 
> For a similar reason, the DOS defences reduce the number of client
> connections to a relay behind a NAT. So we can safely ignore those
> relays for the moment.
> 
> But it would still be useful to talk about the IP address and NAT issue
> in this proposal.
> 

This is not an issue we had considered. For the short term deployment
maybe we can just say not being NAT-ed is a prerequisite for opting in
to measurement.

>> MSM_PARAMS_OK has no payload: it's just padding bytes to make the cell
>> 514 bytes long.
> 
> Should we add ext-type-length-value fields to this cell?
> 
> For example, the MSM_PARAMS_OK cell could be used to communicate the
> relay's recent CPU load and connection load.
> 
>> The payload of MSM_ECHO cells:
>>
>>  - arbitrary bytes [max to fill up 514 byte cell]
> 
> Note:
> 
> Link protocol 3 is still supported, so cells can be 512 or 514 bytes:
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n2094
> 
> Let's say "PAYLOAD_LEN payload (509 bytes)" instead.
> 
> If FlashFlow requires link protocol version 4, let's explain why in
> this proposal.
>

I don't believe FlashFlow requires link protocol 4. Yes the above should
be re-worded.

>> The payload of MSM_BG cells:
>>
>>  - second        [1 byte]
> 
> What are the minimum and maximum valid values for this field?
> 1..msm_duration ?
> 
>>  - sent_bg_bytes [4 bytes]
>>  - recv_bg_bytes [4 bytes]
> 
> What are the minimum and maximum valid values for these fields?
> 
> Should we add ext-type-length-value fields to this cell?
> 
> For example, the MSM_BG cell could be used to communicate the
> relay's current CPU load and connection load.
> 

Indeed we picture MSM_BG cells being used in the longer term to
communicate such information. ext-type-length-value fields are probably
called for.

>> second is the number of seconds since the measurement began. MSM_BG
>> cells are sent once per second from the relay to the FlashFlow
>> coordinator. The first cell will have this set to 1, and each
>> subsequent cell will increment it by one. sent_bg_bytes is the number of
>> background traffic bytes sent in the last second (since the last MSM_BG
>> cell). recv_bg_bytes is the same but for received bytes.
>>
>> The payload of MSM_ERR cells:
>>
>>  - err_code [1 byte]
>>  - err_str  [possibly zero-len null-terminated string]
> 
> We don't have strings in any other tor protocol cells.
> 
> If you need extensible error information, can I suggest using
> ext-type-length-value fields:
> 
>      N_EXTENSIONS     [1 byte]
>      N_EXTENSIONS times:
>         EXT_FIELD_TYPE [1 byte]
>         EXT_FIELD_LEN  [1 byte]
>         EXT_FIELD      [EXT_FIELD_LEN bytes]
> 
> https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n1518
> 
> If strings are necessary, please specify a character encoding
> (ASCII or UTF-8), and an allowed set of characters.
> 
> If we don't whitelist characters, we risk logging terminal escape
> sequences, or other arbitrary data.
> 

I seem to remember strings used between directory authority /directory
mirror relays and clients to communicate certain errors (clock skew?),
but what's probably reality is a *code* is communicated and what I'm
thinking of is merely the Tor client interpreting the code for logging
purposes.

Regardless, we probably don't really need a string. It occurs to me we
might want *something* that carries more information than a code; for
example, a MSM_ERR cell with a code stating "I'm refusing to be measured
because I've been measured too recently" would benefit from a field
stating either time till measurement allowed again or time since last
measurement.

>> The error code is one of:
>>
>>  [... XXX TODO ...]
>>  255 -- OTHER
>>
>> The error string is optional in all cases. It isn't present if the first
>> byte of err_str is null, otherwise it is present. It ends at the first
>> null byte or the end of the cell, whichever comes first.
>>
>> 3.1.4 Measurement Mode
>>
>> The relay considers the measurement to have started the moment it
>> receives the first MSM_ECHO cell from any measurer.
> 
> What happens if the relay never receives a MSM_ECHO cell?
> 
> Do MSM_ECHO cells from invalid measurers count?
> 
> How much of the tor link protocol does the measurer implement?
> Currently, tor requires the following cells:
> * VERSIONS
> * NETINFO
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n542
>

If the relay never receives a MSM_ECHO cell, it never enters measurement
mode (thus it never limits background traffic), eventually times out on
waiting for the measurement to start, sends MSM_ERR cells to connected
measurers/coordinator, and cleans up.

Only measurers that are a part of this measurement can send MSM_ECHO
cells. Other measurers shouldn't have even been allowed to connect.

Measurers don't implement the tor link protocol because they require a
Tor client to do the hard work for them, similar to how I clarified
coordinators above.

>> At this point, the
>> relay
>>
>>  - Starts a repeating 1s timer on which it will report the amount of
>>    background traffic to the coordinator over the coordinator's
>>    connection.
>>  - Enters "measurement mode" and limits the amount of background
>>    traffic it handles according to the torrc option/consensus
>>    parameter.
>>
>> The relay decrypts and echos back all MSM_ECHO cells it receives on
>> measurement connections
> 
> Are MSM_ECHO cells relay cells?
> How much of the relay protocol does the measurer implement?
> 
> The references to decrypting cells suggest that MSM_ECHO cells are
> relay (circuit-level) cells. But earlier sections suggest that they are
> link cells.
> 
> If they are link cells, what key material is used for decryption?
> How do the measurer and relay agree on this key material?
> 
> If they are relay cells, do they use the ntor handshake?
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n1132
>

MEASURE cells are cells like CREATE, CREATED, RELAY, etc.

MSM_ECHO, MSM_PARAMS, etc. cells are MEASURE cells in the same way
RELAY_BEGIN, RELAY_DATA, RELAY_SENDME, etc. are RELAY cells.

The relay needs to do AES on the MSM_ECHO cells (or for simplicity, the
MSM_ECHO cell payload) like it does AES on relay cells. As for key
material necessary to do that, that's an oversight. We've neglected to
specify how it's derived.

Suggestions? Not a cryptographer, but off the top of my head, the
measurer could simply tell the relay to use $key_i for MSM_ECHO cells on
$connection_i. We just want the CPU load on the relay; we're not after
security properties here (other than verifying the relay is actually
doing the crypto, as discussed elsewhere).

>> until it has reported its amount of background
>> traffic the same number of times as there are seconds in the measurement
>> (e.g. 30 per-second reports for a 30 second measurement). After sending
>> the last MSM_BG cell, the relay drops all buffered MSM_ECHO cells,
>> closes all measurement connections, and exits measurement mode.
> 
> To be more precise here, can we say:
> 
> "the relay drops all inbound and outbound MSM_ECHO cells from measurers
> associated with the completed measurement"
> 
> Can we avoid assuming that there is always only one measurement happening
> at one time?
>

I think it's safe/smart/necessary to assume that, for a given relay,
there is always only zero or one measurements happening.

- Measurements are scheduled s.t. coordinators won't try to measure a
relay at the same time.
- A coordinator trying to start a measurement while another one is
ongoing can simply be sent a MSM_ERR cell stating as such.
- The security arguments behind *3.3.2 Weight Inflation* only make sense
when there is only one measurement at a time.

>> During the measurement the relay targets a ratio of background traffic
>> to measurement traffic as specified by a consensus parameter/torrc
>> option. For a given ratio r, if the relay has handled x cells of
>> measurement traffic recently, Tor then limits itself to y = xr/(1-r)
>> cells of non-measurement traffic this scheduling round. The target will
>> enforce that a minimum of 10 Mbit/s of measurement traffic is recorded
>> since the last background traffic scheduling round to ensure it always
>> allows some minimum amount of background traffic.
> 
> Do you mean "a maximum of 10 Mbit/s of measurement traffic" ?
> 

No. When getting ready to handle background traffic, if there has been
less than 10 Mbit/s of measurement traffic recently, Tor will limit
background traffic as if there was indeed 10 Mbit/s of measurement traffic.

This way the relay can always send at least some background traffic, and
a malfunctioning/malicious FlashFlow deployment cannot stop all
background client traffic going through a relay for 30 seconds by not
sending it (very much) measurement traffic.

>> 3.2 FlashFlow Components
>>
>> The FF coordinator and measurer code will reside in a FlashFlow
>> repository separate from little-t tor.
>>
>> There are three notable parameters for which a FF deployment must choose
>> values. They are:
>>
>>  - The number of sockets, s, the measurers should open, in aggregate,
>>    with the target relay. We suggest s=160 based on the FF paper.
>>  - The bandwidth multiplier, m. Given an existing capacity estimate for
>>    a relay, z, the coordinator will instruct the measurers to, in
>>    aggregate, send m*z Mbit/s to the target relay. We recommend m=2.25.
>>  - The measurement duration, d. Based on the FF paper, we recommend
>>    d=30 seconds.
> 
> Are these parameters per-coordinator, or network-wide?
> 
> How are they kept in sync between the coordinator and measurers?
> 

Per-coordinator. The coordinator tells the measurers the parameters for
each measurement.

>> The rest of this section first discusses notable functions of the
>> FlashFlow coordinator, then goes on to discuss FF measurer code that
>> will require supporting tor code.
>>
>> 3.2.1 FlashFlow Coordinator
>>
>> The coordinator is responsible for scheduling measurements, aggregating
>> results, and producing v3bw files. It needs continuous access to new
>> consensus files, which it can obtain by running an accompanying Tor
>> process in client mode.
> 
> Recent tor versions go dormant when they haven't built circuits for a
> while. There are options that prevent dormancy, but they are only designed
> for interactive applications.
> 
> Is the FlashFlow coordinator going to use tor to implement the tor link
> protocol?
> 
> If the coordinator uses tor, then it can use the same tor client instance
> that's downloading its consensuses.
> 
> Otherwise, you might just be better using a small stem script, and a
> download timer.
> 
> If you use a timer, you can download each new consensus, shortly after
> it is created. (Clients often have consensuses that are 1-2 hours old,
> unless specifically configured to fetch from directory authorities.
> Even then, they can take up to an hour to download a new consensus.)
>

As described elsewhere, the coordinator uses a Tor client in order to
avoid implementing the tor link protocol itself. If there is not already
a way to make a Tor client download every new consensus (e.g. a torrc
option or an hourly control port command), we'll want to add that.

>> The coordinator has the following functions, which will be described in
>> this section:
>>
>>  - result aggregation.
>>  - schedule measurements.
>>  - v3bw file generation.
>>
>> 3.2.1.1 Aggregating Results
>>
>> Every second during a measurement, the measurers send the amount of
>> verified measurement traffic they have received back from the relay.
>> Additionally, the relay sends a MSM_BG cell each second to the
>> coordinator with amount of non-measurement background traffic it is
>> sending and receiving.
> 
> What happens if some of these cells is dropped by the relay, due to a
> traffic overload?
> 
> If these cells are exempt from the [Relay]Bandwidth{Rate,Burst} options,
> let's say that in this proposal.
> 
> What happens if some of these cells are delayed due to the MSM_ECHO
> cells?
> 
> How long a delay does the coordinator tolerate?
> 

It would be bad for these cells to get dropped. I can't say if that
means they need to be exempt from BandwidthRate (etc.) options.

IMO it would be fine if the cells were extremely delayed and they all
arrived at the coordinator at the very end of the measurement in a
bunch. Though not ideal, it would be "fine" if they arrived out of order
or a few were lost. They can be reordered fine, and losing a few "just"
leads to inaccuracy.

Obviously I think this should be mitigated. 100%. I'm just saying it's
not a measurement failure if these things happen.

>> For each second's reports, the coordinator sums the measurer's reports.
>> The coordinator takes the minimum of the relay's reported sent and
>> received background traffic. If, when compared to the measurer's reports
>> for this second, the relay's claimed background traffic is more than
>> what's allowed by the background/measurement traffic ratio, then the
>> coordinator further clamps the relay's report down. The coordinator adds
>> this final adjusted amount of background traffic to the sum of the
>> measurer's reports.
>>
>> Once the coordinator has done the above for each second in the
>> measurement (e.g. 30 times for a 30 second measurement), the coordinator
>> takes the median of the 30 per-second throughputs and records it as the
>> estimated capacity of the target relay.
>>
>> 3.2.1.2 Measurement Schedule
>>
>> The short term implementation of measurement scheduling will be simpler
>> than the long term one due to (1) there only being one FlashFlow
>> deployment, and (2) there being very few relays that support being
>> measured by FlashFlow. In fact the FF coordinator will maintain a list
>> of the relays that have updated to support being measured and have opted
>> in to being measured, and it will only measure them.
>>
>> The coordinator divides time into a series of 24 hour periods, commonly
>> referred to as days. Each period has measurement slots that are longer
>> than a measurement lasts (30s), say 60s, to account for pre- and
>> post-measurement work. Thus with 60s slots there's 1,440 slots in a
>> day.
>>
>> At the start of each day the coordinator considers the list of relays
>> that have opted in to being measured. From this list of relays, it
>> repeatedly takes the relay with the largest existing capacity estimate.
>> It selects a random slot. If the slot has existing relays assigned to
>> it, the coordinator makes sure there is enough additional measurer
>> capacity to handle this relay. If so, it assigns this relay to this
>> slot. If not, it keeps picking new random slots until one has sufficient
>> additional measurer capacity.
> 
> What if the coordinator doesn't have enough capacity to handle all the
> relays on the network? (That is, what if all the slots are full?)
>

We can adjust the definition of the measurement period to: the maximum of

1. 24 hours, and
2. the amount of time the FlashFlow deployment with least capacity will
take to measure the entire network + some factor.

If new relays appear during the day and all slots have been filled,
that's unfortunate but they will just wait till the next day.

> What if the capacity is limited at some other point on the internet?
> 
> For example:
> * an intermediate transit provider between the measurer and all the chosen
>   relays
> * the chosen relays are all on the same local network
>

Ideally a single FlashFlow deployment's measurers are diverse to help
mitigate the first point.

For the second, I don't have a good idea at this time. That shouldn't
happen regularly. It will happen sometimes though, so perhaps this
motivates a modification in how the coordinator chooses the weight for a
relay. Instead of the result of the latest measurement, perhaps the
highest result from the last X measurements.

>> Relays without existing capacity estimates are assumed to have the 75th
>> percentile capacity of the current network.
>>
>> If a relay is not online when it's scheduled to be measured, it doesn't
>> get measured that day.
> 
> Online in the consensus, or listening via its ORPort?
> (There's a delay of up to 3 hours here, whenever the relay goes up or
> down.)
> 
> What bandwidth weight does an offline relay get?
> sbws has had issues because it drops offline relays.
>

Online as in both, I think.

I'm not up to speed or have forgotten why continuing to give weight to
offline relays is important (and this may not be the place to enlighten
me). Naively I'd say zero. Assuming that's stupid, I **think** whatever
weight FlashFlow would give it were it online is smarter than some
minimum weight value. Suggestions?

>> 3.2.1.2.1 Example
>>
>> ...
>>
>> 3.2.1.3 Generating V3BW files
>>
>> Every hour the FF coordinator produces a v3bw file in which it stores
>> the latest capacity estimate for every relay it has measured in the last
>> week. The coordinator will create this file on the host's local file
>> system. Previously-generated v3bw files will not be deleted by the
>> coordinator.
> 
> Seems risky, we've seen Torflow fail in the past, because it filled up
> the disk with bandwidth files.
> 
> What's the required disk capacity for a few years of bandwidth files?
>

We can ship a script or provide a parameter to keep the last X v3bw
files if that would be preferable to relying on bwauths using logrotate
themselves or otherwise finding an archival/deletion strategy that fits
their needs.

>> A symbolic link at a static path will always point to the
>> latest v3bw file.
>>
>>    $ ls -l
>>    v3bw -> v3bw.2020-03-01-05-00-00
>>    v3bw.2020-03-01-00-00-00
>>    v3bw.2020-03-01-01-00-00
>>    v3bw.2020-03-01-02-00-00
>>    v3bw.2020-03-01-03-00-00
>>    v3bw.2020-03-01-04-00-00
>>    v3bw.2020-03-01-05-00-00
> 
> You might want to reference the v3bw spec here:
> https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt
> 
>> 3.2.2 FlashFlow Measurer
>>
>> The measurers take commands from the coordinator
> 
> The command protocol is not specified in this proposal.
> 
> For example, does the coordinator send the IPv4 and IPv6 addresses of
> the relay to the measurers?
> 
> Which deployment parameters are sent via the protocol, and which are
> hard-coded in configurations?
>

A Tor proposal did not seem the place for some of these protocols,
options, etc. existing entirely outside little-t tor. We can certainly
elaborate better if that's wrong.

To answer these specific questions: the coordinator would send
fingerprints to the measurers, and the ~only config options the
measurers will have is information regarding the coordinator from which
they shall expect commands. All other FlashFlow options (e.g.
measurement duration) are configured at the coordinator and the coord
informs the measurers.

>> connect to target
>> relays with many sockets, send them traffic, and verify the received
>> traffic is the same as what was sent. Measurers need access to a lot of
>> internal tor functionality. One strategy is to house as much logic as
>> possible inside an compile-time-optional control port module that calls
>> into other parts of tor. Alternatively FlashFlow could link against tor
>> and call internal tor functions directly.
>>
>> [XXX for now I'll assume that an optional little-t tor control port
>> module housing a lot of this code is the best idea.]
> 
> Yes, please don't depend on internal, unspecified interfaces.
> 
>> Notable new things that internal tor code will need to do on the
>> measurer (client) side:
>>
>>  1. Open many TLS+TCP connections to the same relay on purpose.
>>  2. Verify echo cells.
>>
>> 3.2.2.1 Open many connections
>>
>> ...
>>
>> 3.3 Security
>>
>> ...
>>
>> 4. FlashFlow measurement system: Medium term
>>
>> The medium term deployment stage begins after FlashFlow has been
>> implemented and relays are starting to update to a version of Tor that
>> supports it.
> 
> We avoid using tor versions to detect relay features. Instead, we use
> subprotocol versions:
> 
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n2041
> 
> In the first tor release that supports the medium-term FlashFlow, let's
> reserve a "Link" protocol version:
> 
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n2094
> 
> If any of the FlashFlow cells are relay cells, let's also reserve a
> "Relay" protocol version:
> 
> https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n2122
> 
> (We don't want to pick the exact version numbers yet. Let's wait until
> the actual tor release.)
> 
>> We plan to host a FlashFlow deployment consisting of a FF coordinator
>> and a single FF measurer on a single 1 Gbit/s machine. Data produced by
>> this deployment will be made available (semi?) publicly, including both
>> v3bw files and intermediate results.
> 
> All directory authorities publish v3bw files at a standard URL, so if
> you use these files in voting, they will be public.
> 
>> Any development changes needed during this time would go through
>> separate proposals.
>>
>> 5. FlashFlow measurement system: Long term
>>
>> In the long term, finishing-touch development work will be done,
>> including adding better authentication and measurement scheduling, and
>> experiments will be run to determine the best way to integrate FlashFlow
>> into the Tor ecosystem.
>>
>> Any development changes needed during this time would go through
>> separate proposals.
>>
>> 5.1 Authentication to Target Relay
>>
>> Short term deployment already had FlashFlow coordinators using TLS
>> certificates when connecting to relays, but in the long term, directory
>> authorities will vote on the consensus parameter for which coordinators
>> should be allowed to perform measurements. The voting is done in the
>> same way they currently vote on recommended tor versions.
>>
>> FlashFlow measurers will be updated to use TLS certificates when
>> connecting to relays too. FlashFlow coordinators will update the
>> contents of MSM_PARAMS cells to contain measurer TLS certificates
>> instead of IP addresses, and relays will update to expect this change.
> 
> You'll want another new "Link" protocol version for this feature. And
> another type of link specifier.
> 
>> 5.2 Measurement Scheduling
>>
>> Short term deployment only has one FF deployment running. Long term this
>> may no longer be the case because, for example, more than one directory
>> authority decides to adopt it and they each want to run their own
>> deployment. FF deployments will need to coordinate between themselves
>> to not measure the same relay at the same time, and to handle new relays
>> as they join during the middle of a measurement period (during the day).
>>
>> The following is quoted from Section 4.3 of the FlashFlow paper.
>>
>>    To measure all relays in the network, the BWAuths periodically
>>    determine the measurement schedule. The schedule determines when and
>>    by whom a relay should be measured. We assume that the BWAuths have
>>    sufficiently synchronized clocks to facilitate coordinating their
>>    schedules. A measurement schedule is created for each measurement
>>    period, the length p of which determines how often a relay is
>>    measured. We use a measurement period of p = 24 hours.
>>
>>    To help avoid active denial-of-service attacks on targeted relays,
>>    the measurement schedule is randomized and known only to the
>>    BWAuths. Before the next measurement period starts, the BWAuths
>>    collectively generate a random seed (e.g. using Tor’s
>>    secure-randomness protocol). Each BWAuth can then locally determine
>>    the shared schedule using pseudorandom bits extracted from that
>>    seed.
> 
> As noted above, communication between BWAuths reduces their independence,
> and adds additional risk and complexity in the protocol.
> 

All of this is supposed to be non-interactive, yes. It is
under-specified at this time (IMO). I think this is okay for now because
this specifically is a long term thing currently far away.

> Once-Off Shared Secret Exchange
> 
> Here's an alternative protocol, that does not require an additional
> shared random implementation:
> 
> 1. The BWAuths manually exchange a shared secret key (SHARED_SECRET)
>    out-of-band
> 2. Every day, the BWAuths independently derive a shared secret seed for
>    the measurement protocol, using a hash function (H), and tor's public
>    shared random value (SRV):
> 
>       DAILY_SECRET = H(SHARED_SECRET | SRV)
> 
> We might also want to use the period number here, like the SRV and onion
> service hash ring specs.
> 
> The shared secret key should be rotated:
>   * each time a new BWAuth is added or removed from the network, and
>   * 1 year after the last rotation.
> 
> The key rotation can be performed over a few days, because:
> * each BWAuth has one of two keys: the new key, or the old key,
> * overlaps should be rare in practice,
> * when there is an overlap, at most two BWAuths will overlap,
>   one from each key,
> * overlaps have a low impact for most relays.
> 
>>    The algorithm to create the schedule considers each
>>    measurement period to be divided into a sequence of t-second
>>    measurement slots. For each old relay, slots for each BWAuth to
>>    measure it are selected uniformly at random without replacement
>>    from all slots in the period that have sufficient unallocated
>>    measurement capacity to accommodate the measurement. When a new
>>    relay appears, it is measured separately by each BWAuth in the first
>>    slots with sufficient unallocated capacity. Note that this design
>>    ensures that old relays will continue to be measured, with new
>>    relays given secondary priority in the order they arrive.
> 
> It's unclear whether this protocol is interactive or not.
> 
> Here's a protocol that is explicitly non-interactive:
> 
> 1. Measurers are assigned a daily order, based on each coordinator's
>    certificate hash, and the current DAILY_SECRET.
> 2. For each coordinator, in the daily order:
>    a. Relays in a chosen consensus choose a slot at random, based on
>       the DAILY_SECRET, the relay key, and the iteration number
>    c. If another coordinator is already measuring that relay in that
>       slot, increase the iteration number, and repeat from a.
>    b. If the slot is full for the current coordinator, increase the
>       iteration number, and repeat from a.
>    d. Otherwise, allocate that relay to that slot, for that
>       coordinator.
> 
> We might also want to use other shared data here, like the consensus
> timestamp.
> 
> To make sure all the coordinators have the same consensus, we
> should keep a copy of the most recent shared consensus. Here's how
> we can select a shared consensus:
>   * if we're using a scheduled fetch, a consensus from at least 1 hour
>     ago (usually 2300 UTC),
>   * if we're using a tor client to fetch, a consensus from at least 3
>     hours ago (usually 2100 UTC).
> 
> If there isn't a consensus for that time, we should keep the most
> recent consensus before that time.
> 
> It doesn't actually matter if the consensus is a little out of sync,
> most relays will have the same fingerprints, and end up in the same
> slots.
> 
>> 5.3 Experiments
>>
>>   [XXX todo]
>>
>> 5.4 Other Changes/Investigations/Ideas
>>
>> ...
>>
>> 6. Citations
>>
>> [0] F. Thill. Hidden Service Tracking Detection and Bandwidth Cheating
>>    in Tor Anonymity Network. Master’s thesis, Univ. Luxembourg, 2014.
>> [1] A. Johnson, R. Jansen, N. Hopper, A. Segal, and P. Syverson.
>>    PeerFlow: Secure Load Balancing in Tor. Proceedings on Privacy
>>    Enhancing Technologies (PoPETs), 2017(2), April 2017.
>> [2] Mike Perry: Graph onionperf and consensus information from Rob's
>>    experiments https://trac.torproject.org/projects/tor/ticket/33076
> 
> T
>