[tor-dev] Karsten's status report June 1--30

[Karsten Loesing]

Hi all,

below's what I did in June.  Yes, there's still one more day left in
June, but I don't expect to do anything useful tomorrow other than
packing for Italy and pretending to have a half weekend.

Best,
Karsten


Fixed a few problems with the metrics website: bridge usage statistics
graphs were missing 85% of reported bridge statistics due to a truncated
file containing raw observations (#6064); download times in Torperf
graphs were consistently 1 second too high because of a trivial math bug
(#6070); some direct usage graphs showing possible censorship events
displayed upturns or downturns for every single data point (first part
of #6170); direct usage graphs of countries with little usage displayed
missing values where they should display zero users (second part of #6170).

Added a new graph to the metrics website that shows number of cloud
bridges over time (#5629) [0].  Runa is still investigating the sudden
drops in March and April, but it looks like they're related to a problem
with updating cloud images.

Visualized Torperf phases from starting a download to 1) receiving the
CONNECTED cell, 2) receiving the first byte, and 3) receiving the last
byte of the response (#6120).

Looked more into improved bridge usage statistics (#5807) with little
outcome so far.  The directory request statistics reported by bridges
still look very promising for estimating daily bridge users.

Tried to find reasons for missing extra-info descriptors (#3266).  There
are some interesting patterns, but it's still unclear what's going on.

Finished making the new combined Torperf data format the new default
format that we provide on the metrics website (#3036) [1, 2].

Added support for parsing GetTor statistics files to metrics-lib and
made the metrics website code use that library instead of implementing
its own parser.

Discussed sanitizing pluggable transport lines in bridge extra-info
descriptors with George and implemented the required code in metrics-db
(#4957).

Deployed George's changes to the censorship detector script to send out
daily mails for censorship events [3].  Apparently, there are still
plenty of false positives on that list, but the newly gained awareness
might help improving the censorship detector script in the future.

Fixed a problem in DocTor, the consensus-health checker, which looked at
old consensuses for some checks when it shouldn't.

Helped prepare the metrics host for a downtime of a few hours for
getting its RAID card replaced.  Looks like replacing the broken card
has solved previously existing performance problems quite well.

Made a few smaller extensions to Onionoo, the data provider behind
Atlas: added bridge nicknames; added support for searching by country
code (#5960); included exit addresses in summary and search (#5251);
added support for relays and bridges with multiple OR addresses (#5245);
allowed searches for more than 1 search term (#5248); added reverse DNS
lookup results (#5247).  Also wrote a design document for Onionoo [4] to
facilitate writing a Python version of it.

[0] https://metrics.torproject.org/network.html#cloudbridges

[1] https://metrics.torproject.org/data.html#performance

[2] https://metrics.torproject.org/formats.html#torperf

[3]
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-censorship-events

[4] https://gitweb.torproject.org/onionoo.git/blob/HEAD:/DESIGN