[metrics-bugs] #16843 [Metrics/Onionoo]: Add all bwauth measurements (from votes)

[Tor Bug Tracker & Wiki]

#16843: Add all bwauth measurements (from votes)
------------------------------+------------------------------
 Reporter:  cypherpunks       |          Owner:  metrics-team
     Type:  enhancement       |         Status:  assigned
 Priority:  Medium            |      Milestone:
Component:  Metrics/Onionoo   |        Version:
 Severity:  Normal            |     Resolution:
 Keywords:  tor-bwauth-needs  |  Actual Points:
Parent ID:  #24834            |         Points:
 Reviewer:                    |        Sponsor:
------------------------------+------------------------------

Comment (by karsten):

 Replying to [comment:6 teor]:
 > Do you need more developer time, more disk, more CPU, or more RAM?
 > Because we can try to make these things happen.

 So, I wrote a quick hack of an Onionoo that downloads and reads votes in
 addition to all the other descriptors. Here's what I learned:

  - The local descriptor cache without votes is 569M. With votes it's 3.3G.
 That's an increase of 494%!
  - The time to read 3 days of descriptors in the hourly run without votes
 is 10 minutes. With votes it's 16 minutes. That's an increase of 60%. (But
 we typically only read 1 hour of descriptors per hour.)

 So, I'd say we'd need these things to make this happen:
  - Some more disk space on the Onionoo hosts, but not really much more RAM
 or CPUs. (Reading descriptors is still done in a single thread, so reading
 more descriptors simply takes more time.)
  - Some developer time to write a specification patch for this new Onionoo
 feature, an implementation, and tests.
  - Some review time.

 '''teor''': Would you want to work on that specification patch, so that we
 have a better idea what we're supposed to build here?

 If we want to do this properly, we should also resolve a few related
 issues that become a bit more important by adding this feature:
  - Ability to use compression when downloading descriptors from CollecTor
 using metrics-lib (no ticket yet)
  - Ability to process descriptors in parallel using metrics-lib (#21365 is
 related)
  - Ability to handle large descriptor files in metrics-lib (#20395)

 > (Another alternative for #24834 is that we build a quick stem script to
 export the data we need, and import it into Relay Search.)

 For a short-term thing, sure. But long-term this sounds like a maintenance
 nightmare. (After all, Onionoo is the tool to export the data we need and
 import it into Relay Search.)

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/16843#comment:7>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online