[tor-bugs] #2763 [Metrics]: Do we collect descriptors that don't get into the consensus?

[Tor Bug Tracker & Wiki]

#2763: Do we collect descriptors that don't get into the consensus?
---------------------+------------------------------------------------------
 Reporter:  arma     |          Owner:  karsten 
     Type:  task     |         Status:  assigned
 Priority:  normal   |      Milestone:          
Component:  Metrics  |        Version:          
 Keywords:           |         Parent:          
   Points:           |   Actualpoints:          
---------------------+------------------------------------------------------
Changes (by karsten):

  * status:  new => assigned


Comment:

 Replying to [comment:4 nickm]:
 > I wonder if this approach might be insufficient for your requirements.
 It will tell you about descriptors that the authorities 'they have
 accepted and have decided to keep.  It ''won't'' tell us about descriptors
 that the authorities immediately rejected, or ones that they decided (for
 whatever reason) to drop or replace.
 >
 > Do we care about those factors?

 That's a fine question.  I can't say.  I guess Sebastian or arma have an
 answer.  From a metrics POV, we're only interested in the descriptors that
 are referenced from consensuses and maybe votes.  But I understand the
 need to collect unreferenced descriptors for debugging purposes.

 What reasons are there for an authority to reject or drop a descriptor?
 a) unable to parse and b) changes are cosmetic come to mind.  I'm somewhat
 concerned about a) here.  If we want to include descriptors that the
 directory authorities cannot parse, I'll have to improve the metrics code
 for parsing descriptors.  I'd prefer to not include descriptors from case
 a), though.  Descriptors from case b) should be fine to archive.  Are
 there other reasons for the authorities to drop or reject descriptors?

 > As for the information about download size: you can make it much
 smaller.  First, instead of downloading "all", download "all.z".

 Right.  We should do that for all downloads, I guess.

 > Second, instead of downloading all extra-info descriptors, read through
 the descriptors in tor/server/all.z to see which ones you are missing, and
 download only those.  I'd bet these approaches combined would save 60-80%
 of the expected download size.

 Okay, that should work.  Is once per day enough?

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/2763#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online