[tor-dev] Proposal: Expose raw bwauth votes

[teor]

> On 12 Dec 2017, at 09:41, Tom Ritter <tom at ritter.vg> wrote:
> 
> I'm not sure, but I think
> https://trac.torproject.org/projects/tor/ticket/21377 needed a
> proposal so I tried to write one up.

Thanks Tom!

Proposals are great for new directory authority features, particularly
when we're changing the directory specification.

I've quoted the proposal below with my comments:

> Filename: xxx-expose-bwauth_votes.txt
> Title: Have Directory Authorities expose raw bwauth vote documents
> Author: Tom Ritter
> Created: 11-December-2017
> Status: Open
> 
> 1. Introduction
> 
> Bandwidth Authorities (bwauths) perform scanning of the Tor Network
> and calculate observed speeds for each relay. They produce a 'bwauth
> vote file' that is given to a Directory Authority. The Directory
> Authority uses the speed value from this file in its vote file
> denoting its view of the speed of the relay.
> 
> After collecting all of the votes from other Authorities, a consensus
> is calculated, and the consensus's view of a relay's speed is
> determined by choosing the low-median value [or is it high-median?]

low-median

> of all the authorities' values for each relay.
> 
> Only a single metric from the bwauth vote file is exposed by a 
> Directory Authority's vote, however the original file contains
> considerably more diagnostic information about how the bwauth arrives
> at that measurement for that relay.
> 
> 2. Motivation
> 
> The bwauth vote file contains more information that is exposed in the
> overall vote file. This information is useful to debug anomalies in
> relays' utilization and suspected bugs in the (decrepit) bwauth code.
> 
> Currently, all bwauths expose the raw vote file through various (non-
> standard) means, and that file is downloaded (hourly) by a single person
> (as long as his home internet connection and home server is working)
> and archived (with a small amount of robustness.)  
> 
> It would be preferable to have this exposed in a standard manner.
> Doing so would no longer require bwauths to run HTTP servers to expose
> the file, no longer require them to take additional manual steps to
> provide it, and would enable public consumption by any interested
> parties.  We hope that Collector will begin archiving the files.
> 
> 3. Specification
> 
> An authority SHOULD publish the bwauth vote used to calculate its
> current vote. It should make the bwauth vote file available at the

SHOULD

> same time as its normal vote file.

Since the file is created by an external process, and we never store a
copy of any old files, I suggest that we make it available at the URL at
all times, as "the file that the authority has right now".

(And return a 404 when the file does not exist, is not readable, etc.)

> It should make the file available
> at
>   http://<hostname>/tor/status-vote/next/bwauth.z

We shouldn't use next/ unless we're willing to cache a copy of the file
we actually used to vote. If we do, we should serve it from next/ as
soon as we vote using it, then serve it from current/ as soon as the
consensus is created.

If we don't store a copy of the file, we should use a different URL,
like status-vote/now/bwauth, and recommend that the file is fetched at
hh:50, when the votes are created. This would allow us to implement
current/bwauth and next/bwauth in a future version.

Have you thought about versioning the URL if we have multiple flavours
of bwauth file? We could use bwauth-<FLAVOR> like consensuses.

Also, Tor has new compression options for zstd and lzma.

Given that this is an externally-controlled file, we could stream it
from disk and compress it on the fly with something cheap like gzip
or zstd.

> It MUST NOT attempt to send its bwauth vote file in a HTTP POST to
> other authorities and it SHOULD NOT make bwauth vote files from other
> authorities available.
> 
> 4. Security Implications
> 
> The raw bwauth vote file does not [really: is not believed to] expose
> any sensitive information.  All authorities currently make this
> document public already, an example is at
>   https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile

How large is the file?
Maybe we should pre-compress it, to avoid CPU exhaustion attacks.
If we did this, we could say that it's safe, as long as it is smaller
than the full consensus or full set of descriptors.

> 5. Compatibility
> 
> Exposing the document presents no compatibility concerns.
> 
> The compatibility concern is with applications that want to consume
> the document. The bwauth vote file has no specification, and has been
> extended in ad-hoc ways. Applications that merely wish to archive the
> document (e.g. Collector) won't have a problems. Applications that
> want to parse it may encounter errors if a new (unexpected) field is
> added, or assumptions are made about the text encoding or formatting
> of the document.

Parsers may also encounter issues if we transition to another bwauth
implementation. We could remove fields if this happens.

T

-- 
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
------------------------------------------------------------------------