[tor-bugs] #11648 [Tor]: Problem parsing .z-compressed descriptors fetched via DirPort

[Tor Bug Tracker & Wiki]

#11648: Problem parsing .z-compressed descriptors fetched via DirPort
-------------------------+--------------------------------
     Reporter:  karsten  |      Owner:
         Type:  defect   |     Status:  needs_review
     Priority:  normal   |  Milestone:  Tor: 0.2.5.x-final
    Component:  Tor      |    Version:
   Resolution:           |   Keywords:  tor-relay
Actual Points:           |  Parent ID:
       Points:           |
-------------------------+--------------------------------

Comment (by wfn):

 Replying to [comment:23 nickm]:
 > In that case, I have a theory that my branch "bug11648" will fix this.
 I'm still curious where the fingerprints for the missing relays are coming
 from in the all.z case, though.

 TL;DR patch works, and seems to work in the very way that we'd assumed it
 should work. At this point I don't think there's any doubt how strange
 `.z`s are produced: if the {microdescriptor, descriptor} for the last
 digest on a list of digests to be processed cannot be fetched, a needed
 call to close the zlib stream is skipped. The question (maybe) remains
 ''why'' a digest list for `all.z` may contain a non-fetchable entry at the
 end ''so frequently'' (this is empirical, and unclear.)

 As far as the patch is concerned, IMO it's good. (See below for details.)

  * Can't (yet) reproduce bogus (with `0000ffff` in the end) `all.z` on a
 directory mirror
  * Can reproduce bogus "unfetchable descriptor at the end" archive:
     * `curl -vv
 http://95.85.10.71/tor/server/d/f53bcf37a5233f4bf91373a62f92692a526e866d+1917db60e6c769567fc364e9dbb6fe8aa5feeb16.z
 | xxd | tail`
   * This archive is not bogus (parsed by python's zlib; no `0000ffff`
 (`Z_SYNC_FLUSH` end)) when running Nick's patch - on a different router:
     * `curl -vv
 http://5.135.188.202:995/tor/server/d/f53bcf37a5233f4bf91373a62f92692a526e866d+1917db60e6c769567fc364e9dbb6fe8aa5feeb16.z
 | xxd | tail`
   * Whereas existing+existing.z good for both:
     * `curl -vv
 http://95.85.10.71/tor/server/d/f53bcf37a5233f4bf91373a62f92692a526e866d+8f6ff348d3a6dd5ce1fc93d030a4c82acb842c6e.z
 > vicare_good.z`
     * `curl -vv
 http://5.135.188.202:995/tor/server/d/f53bcf37a5233f4bf91373a62f92692a526e866d+8f6ff348d3a6dd5ce1fc93d030a4c82acb842c6e.z
 > ravine_good.z`
     * `diff vicare_good.z ravine_good.z` => identical (and good)
     * (note that descriptors may expire etc.; later on, you may need to
 supply different digests)

   *
 [https://github.com/wfn/tor/commit/b0d64a3cf52f01002a5519c5ceb10e8ea1370691
 Inserted a couple of `log_info()`s] (branch
 "bug11648_dirserv_log_bogus_desc") (the "log_info() if descriptor or
 microdescriptor lookup fails" might as well eventually go into master IMO,
 because IMO it's `log_info()`-worthy. Unless logging router digests
 requested at INFO level is not a good idea?)
     * "processed last digest" not logged when last digest lookup fails
 (confirms one of the assumptions)
     * "processed last digest" logged when last digest lookup succeeds
     * "failed to look up descriptor" logged as intended

 Directory mirror 5.135.188.202:995 running
 [https://github.com/wfn/tor/commits/bug11648_dirserv_log_bogus_desc
 "bug11648_dirserv_log_bogus_desc"] (which is on top of
 [https://gitweb.torproject.org/nickm/tor.git/shortlog/refs/heads/bug11648
 "bug11648"]), directory mirror 95.85.10.71:80 running vanilla 0.2.4.21.

   * would be nice to see what kinds of dangling non-fetchable digests are
 on turtles' `all`. Would be possible to do that in a simple manner by
 running those `log_info()`s, enabling INFO log, and grepping for "last
 digest on the stack".
   * still curious why turtles' (and maybe in general) `all.z` contain
 nonexistent digests at the end so often.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/11648#comment:27>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online