[metrics-bugs] #21378 [Metrics/CollecTor]: Archive bwauth bandwidth files

[Tor Bug Tracker & Wiki]

#21378: Archive bwauth bandwidth files
------------------------------------+------------------------------
 Reporter:  tom                     |          Owner:  metrics-team
     Type:  enhancement             |         Status:  new
 Priority:  Medium                  |      Milestone:
Component:  Metrics/CollecTor       |        Version:
 Severity:  Normal                  |     Resolution:
 Keywords:  tor-bwauth tor-dirauth  |  Actual Points:
Parent ID:                          |         Points:
 Reviewer:                          |        Sponsor:
------------------------------------+------------------------------

Comment (by teor):

 Replying to [comment:14 karsten]:
 > I just went through the long discussion above and tried to identify next
 steps. irl's list of needed changes looks pretty good. I'll add some
 thoughts to these steps below that we need to discuss when implementing
 this.
 >
 > Replying to [comment:10 irl]:
 > > It looks like the changes that are needed are:
 > > 1) Teach RelayDescriptorDownloader to download the new URL (in the
 [https://gitweb.torproject.org/collector.git/tree/src/main/java/org/torproject/collector/relaydescs/RelayDescriptorDownloader.java#n685
 downloadDescriptors] function)
 >
 >   - We can either attempt to fetch this file from each authority every
 time, or we can have a config option which authorities should have them.

 I suggest "each authority every time", because a hard-coded config will
 miss some of the bandwidth files on new bandwidth authorities.

 > In the future, we can switch to fetching only those files that are
 referenced from votes, unless for some reason we want to have non-
 referenced files, too.

 How are bandwidth files referenced from votes?

 I don't think we will implement "bandwidth-file-url" from
 https://trac.torproject.org/projects/tor/ticket/21378?replyto=14#comment:6

 Tor 0.3.5? and later add bandwidth file headers to each vote, and we may
 add a bandwidth file hash in future. Once all authorities upgrade, you can
 fetch the bandwidth file if the vote contains headers.

 >   - The relaydescs module runs twice per hour, so it's going to download
 the file twice every hour. Again, if we only fetch referenced files, we
 wouldn't download the same file more than once.

 I am not sure if we plan on implementing "referenced files" in Tor. Can
 you explain what you mean?

 > But it sounds like the initial version will be rather simple in this
 regard. Which is fine.

 I think Juga has written code for a more complex version. But we will
 focus on getting the simple version working first.

 >   - I assume there are no plans that authorities serve bandwidth files
 of other authorities? That's different for votes which are cached by other
 authorities. Should be fine, but something to consider for the future.

 Votes are posted, fetched, and cached by authorities so that each
 authority can create a consensus.
 There's no equivalent for bandwidth files, so we probably won't implement
 bandwidth file caching.
 But if you tell us you really need it, we could work something out.

 > > 2) Teach
 [https://gitweb.torproject.org/collector.git/tree/src/main/java/org/torproject/collector/relaydescs/RelayDescriptorParser.java
 RelayDescriptorParser] to recognise the file
 >
 >  - While we're waiting for #21377, can we have a sample file to start
 writing some parsing code?

 moria1's votes have a bandwidth-file-headers line, but it's based on
 torflow's bandwidth file, so it only has a timestamp:
 http://128.31.0.34:9131/tor/status-vote/current/authority

 The bandwidth file spec contains sample data:
 https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-
 spec.txt#n450

 We're working on version 1.2.0 of the format for sbws 1.0 in #28085. When
 sbws 1.0 is ready, we will update the spec with sample data from the
 latest sbws.

 > > 3) Teach
 [https://gitweb.torproject.org/collector.git/tree/src/main/java/org/torproject/collector/relaydescs/ArchiveWriter.java
 ArchiveWriter] where it should put the files in CollecTor's hierachy
 >
 >  - Let's discuss what should go into the file name. Timestamp,
 fingerprint, and digest? Maybe something similar to the vote file name
 format (with some parts shortened): `2018-11-05-09-00-00-vote-
 EFCBE720[...]-0D97EDB6[...]`?
 >  - As part of this step, we might have to teach metrics-lib to recognize
 the new descriptor type. I believe that CollecTor will store it anyway,
 but it's going to complain loudly. Just in case it acts up, we can teach
 metrics-lib to just recognize the descriptor type without providing
 getters for descriptor contents.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/21378#comment:15>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online