[tor-bugs] #11648 [Tor]: Problem parsing .z-compressed descriptors fetched via DirPort

[Tor Bug Tracker & Wiki]

#11648: Problem parsing .z-compressed descriptors fetched via DirPort
-------------------------+-----------------------
     Reporter:  karsten  |      Owner:
         Type:  defect   |     Status:  new
     Priority:  normal   |  Milestone:
    Component:  Tor      |    Version:
   Resolution:           |   Keywords:  tor-relay
Actual Points:           |  Parent ID:
       Points:           |
-------------------------+-----------------------

Comment (by karsten):

 Without looking at `dirserv.c`, it looks like missing descriptors might be
 causing this problem.  Here's an easy way to reproduce:

 Fetch an ''existing'' descriptor by digest:

 {{{
 $ curl
 http://128.31.0.39:9131/tor/server/d/f53bcf37a5233f4bf91373a62f92692a526e866d.z
 > moria1-f5.z
   % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                  Dload  Upload   Total   Spent    Left
 Speed
 100   874    0   874    0     0    300      0 --:--:--  0:00:02 --:--:--
 300
 }}}

 Attempt to fetch a ''non-existing'' descriptor by digest:

 {{{
 $ curl
 http://128.31.0.39:9131/tor/server/d/1917db60e6c769567fc364e9dbb6fe8aa5feeb16.z
 > moria1-19.z
   % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                  Dload  Upload   Total   Spent    Left
 Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--
 0
 }}}

 So far so good, now fetch both at once:

 {{{
 $ curl
 http://128.31.0.39:9131/tor/server/d/f53bcf37a5233f4bf91373a62f92692a526e866d+1917db60e6c769567fc364e9dbb6fe8aa5feeb16.z
 > moria1-f5+19.z
   % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                  Dload  Upload   Total   Spent    Left
 Speed
 100   875    0   875    0     0   2523      0 --:--:-- --:--:-- --:--:--
 2528
 }}}

 Note in the `curl` output how the second download is 1 byte larger than
 the first.

 Look at the binary diff using `xxd`:

 {{{
 $ diff <(xxd moria1-f5.z) <(xxd moria1-f5+19.z)
 1c1
 < 0000000: 78da 9593 c9ae a338 1486 f73c 85d7 15e5  x......8...<....
 ---
 > 0000000: 78da 9493 c9ae a338 1486 f73c 85d7 15e5  x......8...<....
 55c55
 < 0000360: 9b13 ff28 e307 212c 71aa                 ...(..!,q.
 ---
 > 0000360: 9b13 ff28 e307 0000 00ff ff              ...(.......
 }}}

 There's one difference in the first line, but more importantly, the last
 line differs.  The second download ends with `0x0000ffff` which we saw
 before.  It ''seems'' this was supposed to be the separator between first
 and second descriptor.  But then tor didn't find a second descriptor and
 just closed the stream.

 Now, how's a compressed result of two descriptors supposed to look like?

 Let's fetch another ''existing'' descriptor:

 {{{
 $ curl
 http://128.31.0.39:9131/tor/server/d/6d44fd5b8e5b7382089b02614eddb64b290eed42.z
 > moria1-6d.z
   % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                  Dload  Upload   Total   Spent    Left
 Speed
 100   817    0   817    0     0    598      0 --:--:--  0:00:01 --:--:--
 598
 }}}

 And now we're fetching both ''existing'' descriptors at once:

 {{{
 $ curl
 http://128.31.0.39:9131/tor/server/d/f53bcf37a5233f4bf91373a62f92692a526e866d+6d44fd5b8e5b7382089b02614eddb64b290eed42.z
 > moria1-f5+6d.z
   % Total    % Received % Xferd  Average Speed   Time    Time     Time
 Current
                                  Dload  Upload   Total   Spent    Left
 Speed
 100  1497    0  1497    0     0   1257      0 --:--:--  0:00:01 --:--:--
 1257
 }}}

 Here are the first line, all lines containing `0x0000ffff`, and the last
 line:

 {{{
 0000000: 78da 9493 c9ae a338 1486 f73c 85d7 15e5  x......8...<....
 ...
 0000360: 9b13 ff28 e307 0000 00ff ffb5 d45b 8f9a  ...(.........[..
 ...
 00005d0: 967f 07e8 279b 30cd 28                   ....'.0.(
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/11648#comment:21>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online