[tor-project] Core Tor sponsor4 March report

[isabela]

Core Tor team March 2017 Report:

For March we did a good chunk of work for Proposal 140 [1] which
implements consensus diffs, first we ran again our numbers for it to be
able to see how much bandwidth we expect to save [2]. Then we start to
work on actually change the code to implement the proposal, we did:

   => #21651 - Prop140 / compression: Refactor directory cache spooling
code [3]
   => #21645 - Prop140 / compression: Unified directory cache backend [4]
   => #21644 - Prop140: Fuzz the diff and patch code [5]
   => fix on the performance of SHA3-  keccak. This was caught by a
performance report that showed the function was super slow. By fixing
this we improve it speed 15%. [6]

We finished the implementation of Proposal 274! [7] It will be part of
our 0.3.1 release [8]. This work will reduce the onion key rotation rate
from the current value (7 days) to something closer to 28 days. Doing
this will reduce the total microdescriptor download volume by
approximately 70%.

We continue with our effort to revise, analyze and write proposals for
ways to use less directory bandwidth. One of the ideas [9] was to
explore the possibility of saving bandwidth by improving our compression
system, we want to check if offering other compression mechanisms that
might fit better scenarios where the trade of bandwidth usage for CPU
process makes more sense.

We finished writing Proposal 278 [10], here is an email to tor-dev [11]
where we shared the first draft for feedback from the community, it has
a good problem statement that can help understand the benefits of this
approach, and some information about the analysis we did.

Then we move to work on the actual proposal, below is the work we
started in March but that will be merged in April:
  => #21667: Prop278: Handle new headers in directory.c [12]
  => #21662: prop278: Add support for LZMA2 and/or Zstandard [13]
  => #21663: prop278: Refactor the torgzip module to support additional
compression schemes [14]

At the end of March The Tor Project had its biannual Dev meeting in
Amsterdam. There we had a session with our Metrics team lead, Karsten,
and Nathan from Guardian Project to discuss Directory Bandwidth
Reduction [15].

This meeting was very important as we want to make sure we coordinate
with the Metrics team about the work we are doing since it can influence
how they count things. And we also want to take the opportunity to get
feedback from Mobile, since mobile users with low-bandwidth is one of
our target audience for this work.

After Amsterdam we had a follow up meeting online early this week (4/4)
as well [16].

This is other work we did in March but haven't been merged yet:
  => Needs Revision: #20021 Require ntor-onion-key in microdescriptors,
descriptors [17]
  => Needs Review: #21642 Prop275: Eliminate "published" times from
microdescriptor consensus [18]
  => Needs Review: #21869 Labeled storage backend for consensus docs and
diffs [19]

[1]
https://gitweb.torproject.org/torspec.git/tree/proposals/140-consensus-diffs.txt
[2] https://trac.torproject.org/projects/tor/ticket/21210
[3] https://trac.torproject.org/projects/tor/ticket/21651
[4] https://trac.torproject.org/projects/tor/ticket/21645
[5] https://trac.torproject.org/projects/tor/ticket/21644
[6] https://trac.torproject.org/projects/tor/ticket/21737
[7]
https://gitweb.torproject.org/torspec.git/tree/proposals/274-rotate-onion-keys-less.txt
[8] https://trac.torproject.org/projects/tor/ticket/21641
[9]  https://trac.torproject.org/projects/tor/ticket/21211
[10]
https://gitweb.torproject.org/torspec.git/tree/proposals/278-directory-compression-scheme-negotiation.txt
[11] https://lists.torproject.org/pipermail/tor-dev/2017-March/011959.html
[12] https://bugs.torproject.org/21667
[13] https://bugs.torproject.org/21662
[14] https://bugs.torproject.org/21663
[15]
https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes/Sponsor4
[16] https://pad.riseup.net/p/chikeuBN1OUh
[17] https://trac.torproject.org/projects/tor/ticket/20021
[18] https://trac.torproject.org/projects/tor/ticket/21642
[19] https://trac.torproject.org/projects/tor/ticket/21869