[tor-project] Sponsor4 work completion report from Core Tor team [final report on this]

[isabela]

Core Tor Team August 2017 report

Work completion report:

Subobjective 1.1: Reduce Tor processing overhead for low-bandwidth
scenarios.
Activity: Improve the Directory Authority consensus part of the Tor
network in order to optimize low bandwidth users experience.


We created a plan[1] to achieve this goal, where we would into each idea
we had or could find at our issue tracker or email lists to improve the
Directory Authority consensus and implement the most promising ones.

Since all this is new at Tor, we had to start by building the tools[2,3]
to perform this measurement and these analyses to choose the best ideas
to implement for this activity.

You will find on this report the list of ideas we identified as the best
solutions for this activity goal. We also documented[4] this selection
process on our wiki.

The design changes ideas we choose gave us 45% of bandwidth reduction
for relays and 41% for clients, comparing with the releases prior to the
0.3.1 where these ideas were implemented.

A rock on our road for this project was a bug we reported in May related
to the implementation of proposals 140 and 278, that cost us some time
to figure out. This bug was leading to more bandwidth usage which is
completely the contrary of our goal with this project. The bug was fixed
and documented on ticket #22702 [5].

Design changes implemented to improve Directory Authority consensus part
of Tor network:

Idea: Consensus diffs
​	* Proposal 140 [6]. Ticket #13339 [7].
	* Idea: Instead of fetching a consensus, clients could fetch a diff
from the most recent consensus they have.
	* Status: Implemented in Tor 0.3.1.x.

Idea: Negotiate better compression algorithms
	* Proposal 278 [8]. Ticket #21211 [9].
	* Idea: There are other algorithms like LZMA/LZMA2 or bzip2 that
deliver better compression than zlib. We could use HTTP encoding
negotiation to select them when available.
	* Status : Implemented in Tor 0.3.1.x.

Idea: Rotate onion keys less frequently
	* Proposal: 274. Ticket #21641 [10]
	* Idea: In order to limit the bandwidth needed for microdescriptor
listing and transmission, we reduce the onion key rotation rate from the
current value (7 days) to something closer to 28 days. Doing this will
reduce the total microdescriptor download volume  by approximately 70%.
	* Status: Implemented in Tor 0.3.1.x

Idea: Avoid requesting microdescriptors in small batches
	* Idea: Currently when we want e.g. 21 microdescriptors, we'll probably
request them in 3 batches of 7. This makes our compression ratios worse
than they would be otherwise. But it ensures that a single cache can't
selectively deny us microdescriptors.
	* Status: Ticket #23220 [11] implements a simple version of this.

[1]
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Sponsor4Plan
[2] https://github.com/nmathewson/consensus-diff-analysis
[3] https://gitlab.com/ahf/tor-sponsor4-compression
[4]
https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Sponsor4Design
[5] https://trac.torproject.org/projects/tor/ticket/22702
[6]
https://gitweb.torproject.org/torspec.git/tree/proposals/140-consensus-diffs.txt
[7] https://trac.torproject.org/projects/tor/ticket/13339
[8]
https://gitweb.torproject.org/torspec.git/tree/proposals/278-directory-compression-scheme-negotiation.txt
[9]  https://trac.torproject.org/projects/tor/ticket/21211
[10] https://trac.torproject.org/projects/tor/ticket/21641
[11] https://trac.torproject.org/projects/tor/ticket/23220