[tor-bugs] #9765 [TorDNSEL/TorBEL]: TorDNSel exit lists are missing expected data

[Tor Bug Tracker & Wiki]

#9765: TorDNSel exit lists are missing expected data
---------------------------------+----------------------
     Reporter:  Ry               |      Owner:
         Type:  defect           |     Status:  assigned
     Priority:  normal           |  Milestone:
    Component:  TorDNSEL/TorBEL  |    Version:
   Resolution:                   |   Keywords:  TorCheck
Actual Points:                   |  Parent ID:
       Points:                   |
---------------------------------+----------------------

Comment (by Ry):

 Replying to [comment:1 karsten]:
 > TorCheck had issues in the past months (https://blog.torproject.org/blog
 /tor-check-outage-03-and-04-july-2013), so maybe it's related to that.
 Can you re-run your analysis on the archives of, say, all of 2013?  It
 would be interesting to know whether this is a new problem or not.
 AFAIK that was due to implementation details of the older(still current)
 TorCheck version which wasn't reliant on published exit-lists as much. I
 can surely check across 2013 at the weekend perhaps (Don't really have the
 connection to actually download that much data right this moment)

 -------

 As for the Tor client not seeing the whole consensus, I wasn't exactly
 sure what you were suggesting, so I did two things. For my Tor client
 (built off master), I ran two scripts:

 - https://gist.github.com/Ryman/ee2c11e7107a81c926c8
    Comparing the cached-consensus with the latest consensus on the metrics
 server that I used for the OP.
    There have been no differences for the last two hours for which I
 tried. (If you need to test, there's a comment with a helpful rsync
 command that will pull the latest consensus doc, if you change the
 date/hour)

 - https://gist.github.com/Ryman/1f2e76d7465abb7ea603
    Comparing the fingerprints present in the cached consensus document and
 a merge of both cached-descriptors and cached-descriptors.new. In this
 comparison I have found a really tiny number of servers are missing (3
 when I started testing, seemed to stick at 2 for an hour throughout 2
 different consensus files) If you don't want to compare against .new files
 then supply a -s flag, but I think this would be an error to do so?

 From this, I conclude that the client likely sees enough of the consensus,
 so that the discrepancy in the OP is not caused by that specifically.

 It's likely worth running the scripts against the version of Tor client
 being used by TorDNSEL in production just as a sanity check. I think arma
 might be on the right lines in that it's potentially a problem between
 TorDNSEL and Tor, or TorDNSEL is perhaps timing out connections and never
 retrying/logging them?

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/9765#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online