[tor-project] Core Tor team sponsor4 July report

[isabela]

Core Tor Team July 2017 Report

In July we reviewed, triaged and fixed misc bugs related to releases
0.3.0 and 0.3.1:

* Never send a consensus which the downloader already has. [1]

This was an issue we identified that make the most trouble in test
networks, where the consensus update rate is so fast that relays
frequently try to download a consensus. We investigated and identified
what caused the issue and fixed it.


* LZMA coder causes crash when the sandbox is enabled. [2]

While doing measurements for this project we noticed that Tor instances
running as relays or authorities would sometimes crash when the sandbox
is enabled. This is due to the MALLOC_MP_LIM value in sandbox.c, which
is currently set to 16 MB, being too low. We limit our LZMA coder to
only use 16 MB, but the coder allocates some additional data other than
its internal buffer, which leads to the crash.


* Investigated: Assertion failure in consensus_cache_entry_handle_get on
Windows.[3]

We created a mitigation and diagnostic branch. It doesn't fix this, but
it makes it nonfatal and tries to get more useful info if it happens
again. And receive a report that is still present in 0.3.1.4-alpha. Will
continue investigation in August.

* Bug: src/common/compress.c:576: tor_compress_process: [4]

We fixed a bug found 0.3.1  at a stable Gentoo Linux server.


* Bridge unavailable during differential consensus update [5]

Bug reported for Raspberry Pi running Raspbian. we revised our
threadpool implementation to prevent this background work from affecting
other background work.  There are 3 changelog entries for it in
0.3.1.5-alpha.

** Major bugfixes (relay, performance):
Perform circuit handshake operations at a higher priority than we use
for consensus diff creation and compression. This should prevent
circuits from starving when a relay or bridge receives a new consensus,
especially on lower-powered machines. Fixes bug 22883; bugfix on
0.3.1.1-alpha.

** Minor features (directory cache, consensus diff):
Add a new MaxConsensusAgeForDiffs option to allow directory cache
operators with low-resource environments to adjust the number of
consensuses they'll store and generate diffs from. Most cache operators
should leave it unchanged. Helps to work around bug 22883.

** Minor features (relay, performance):
Always start relays with at least two worker threads, to prevent
priority inversion on slow tasks. Part of the fix for bug 22883.
Allow background work to be queued with different priorities, so that a
big pile of slow low-priority jobs will not starve out higher priority
jobs.This lays the groundwork for a fix for bug 22883.


* zstd tests fail with libzstd 1.3.0 [6]

Write zstd epilogues correctly when the epilogue requires reallocation
of the output buffer, even with zstd 1.3.0. (Previously, we worked on
1.2.0 and failed with 1.3.0).


* consdiff test fails on OS X [7]

test_consdiff_base64cmp would fail on OS X because while OS X follows
the standard of (less than zero/zero/greater than zero), it doesn't
follow the convention of (-1/0/+1). Make the test comply with the standard.


We started to land updates to dir-spec.txt for prop #278 in
GL/ahf/torspec/tree/prop278-in-dir-spec [8] And concluded our review of
lz4 compressor [9]. For that we added a simple LZ4 test to our own
little benchmarking tool [10].

Our results [11] (results) showed that LZ4 is a lot faster than anything
else we have (even beats gzip at compression level 1), but the
compression ratio is also worse than anything else we have for the
cached-consensusdocument (~2.2 MB of structured text).

Since we are currently batch processing the compression of our "larger"
files in the background we decided we wouldn't win much by including LZ4
here.

For August we feel that we are pretty close from finishing this
deliverable and we plan to publish a blog post about compression +
consensus diff work done with this project. Hope to share that soon.


[1] https://trac.torproject.org/projects/tor/ticket/22702
[2] https://trac.torproject.org/projects/tor/ticket/22751
[3] https://trac.torproject.org/projects/tor/ticket/22752
[4] https://trac.torproject.org/projects/tor/ticket/22719
[5] https://trac.torproject.org/projects/tor/ticket/22883
[6] https://trac.torproject.org/projects/tor/ticket/22927
[7] https://trac.torproject.org/projects/tor/ticket/22870
[8] https://trac.torproject.org/projects/tor/ticket/22275
[9] https://trac.torproject.org/projects/tor/ticket/22819
[10] https://gitlab.com/ahf/tor-sponsor4-compression/tree/master/bench
[11]
https://docs.google.com/spreadsheets/d/1WzFXQGfH8yI4WCCRAEQBt1Z_sC-3hMkmE6HNKDaaSiE/edit#gid=0