[tor-bugs] #21378 [Metrics/CollecTor]: Archive bwauth bandwidth files

[Tor Bug Tracker & Wiki]

#21378: Archive bwauth bandwidth files
------------------------------------+------------------------------
 Reporter:  tom                     |          Owner:  metrics-team
     Type:  enhancement             |         Status:  new
 Priority:  Medium                  |      Milestone:
Component:  Metrics/CollecTor       |        Version:
 Severity:  Normal                  |     Resolution:
 Keywords:  tor-bwauth tor-dirauth  |  Actual Points:
Parent ID:                          |         Points:
 Reviewer:                          |        Sponsor:
------------------------------------+------------------------------
Changes (by karsten):

 * cc: metrics-team (added)
 * priority:  Low => Medium


Old description:

> The raw bwauth votes (sample:
> https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile) contain
> information such as last measured time, circuit failures and (eventually)
> scanner information. This can be used for debugging purposes.
>
> Blocked by #21377

New description:

 The raw bwauth votes (sample:
 https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile) contain
 information such as last measured time, circuit failures and (eventually)
 scanner information. This can be used for debugging purposes.

 Blocked by #21377, possible next steps in [#comment:14 comment 14].

--

Comment:

 I just went through the long discussion above and tried to identify next
 steps. irl's list of needed changes looks pretty good. I'll add some
 thoughts to these steps below that we need to discuss when implementing
 this.

 Replying to [comment:10 irl]:
 > It looks like the changes that are needed are:
 > 1) Teach RelayDescriptorDownloader to download the new URL (in the
 [https://gitweb.torproject.org/collector.git/tree/src/main/java/org/torproject/collector/relaydescs/RelayDescriptorDownloader.java#n685
 downloadDescriptors] function)

   - We can either attempt to fetch this file from each authority every
 time, or we can have a config option which authorities should have them.
 In the future, we can switch to fetching only those files that are
 referenced from votes, unless for some reason we want to have non-
 referenced files, too.
   - The relaydescs module runs twice per hour, so it's going to download
 the file twice every hour. Again, if we only fetch referenced files, we
 wouldn't download the same file more than once. But it sounds like the
 initial version will be rather simple in this regard. Which is fine.
   - I assume there are no plans that authorities serve bandwidth files of
 other authorities? That's different for votes which are cached by other
 authorities. Should be fine, but something to consider for the future.

 > 2) Teach
 [https://gitweb.torproject.org/collector.git/tree/src/main/java/org/torproject/collector/relaydescs/RelayDescriptorParser.java
 RelayDescriptorParser] to recognise the file

  - While we're waiting for #21377, can we have a sample file to start
 writing some parsing code?

 > 3) Teach
 [https://gitweb.torproject.org/collector.git/tree/src/main/java/org/torproject/collector/relaydescs/ArchiveWriter.java
 ArchiveWriter] where it should put the files in CollecTor's hierachy

  - Let's discuss what should go into the file name. Timestamp,
 fingerprint, and digest? Maybe something similar to the vote file name
 format (with some parts shortened): `2018-11-05-09-00-00-vote-
 EFCBE720[...]-0D97EDB6[...]`?
  - As part of this step, we might have to teach metrics-lib to recognize
 the new descriptor type. I believe that CollecTor will store it anyway,
 but it's going to complain loudly. Just in case it acts up, we can teach
 metrics-lib to just recognize the descriptor type without providing
 getters for descriptor contents.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/21378#comment:14>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online