[tor-bugs] #5805 [Metrics Data Processor]: Compare anguilla's tarballs to yatei's and maybe merge them

[Tor Bug Tracker & Wiki]

#5805: Compare anguilla's tarballs to yatei's and maybe merge them
------------------------------------+---------------------------------------
 Reporter:  karsten                 |          Owner:  karsten
     Type:  task                    |         Status:  new    
 Priority:  minor                   |      Milestone:         
Component:  Metrics Data Processor  |        Version:         
 Keywords:                          |         Parent:         
   Points:  12                      |   Actualpoints:         
------------------------------------+---------------------------------------
 weasel was running his directory-archive script until a week or two ago.
 I want to compare anguilla's tarballs to yatei's to figure out if yatei is
 missing some descriptors and why, and to merge missing descriptors into
 yatei's tarballs.

 This ticket is mostly here to note down the endless hours that I already
 worked on this task, mostly because I need to write new comparison scripts
 and investigate differences between single descriptors manually before
 identifying a pattern.  As of now, I spent 9 points on this task, and I'm
 not done.  I think another 3 points remain.  The task looked so tiny when
 I decided to do it, but it's also important enough to spend the remaining
 points.

 Current insights from the comparison, which might turn into new tasks,
 are:

  - Quite a few of the consensuses collected by yatei have missing or
 extraneous signatures as compared to anguilla's.  This has to do with
 authorities serving consensuses that don't have all signatures.  I don't
 really care, so I'm probably leaving this alone.

  - Quite often, missing a consensus automatically means missing all votes.
 We might switch to downloading votes by all known authorities, not only by
 the ones contained in a consensus (which we're missing in these cases).
 Not super important, but probably worth doing.

  - We have quite a few files in yatei's tarballs that are empty or
 truncated.  We need to try parsing descriptors with metrics-lib (which is
 not yet used by metrics-db) and only store valid descriptors to disk.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/5805>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online