[tor-project] OONI Monthly Report: December 2019

[Maria Xynou]

Hello Tor world,

In December 2019, the OONI team prioritized on the following:

* Integrating the new OONI Probe test for Psiphon into the OONI Probe
desktop app
* Provisioning of Psiphon configuration into apps
* API query optimization
* Reducing technical debt and bug fixing

## Circumvention tool testing

Throughout December 2019, we made important progress on the testing of
circumvention tools.

Specifically, we:

* Added support for giving out Psiphon configuration files to clients in
orchestra: https://github.com/ooni/orchestra/issues/78
* Added client support for performing authenticated orchestra requests:
https://github.com/ooni/probe-engine/issues/109
* Added support for running circumvention tool tests inside of the OONI
Probe CLI: https://github.com/ooni/probe-cli/issues/77

Through the above work, we laid the foundation for implementing future
circumvention tool tests inside of OONI Probe.

You can now download and run the new OONI Probe Psiphon test via one of
the latest ooni/probe-cli release candidate builds:
https://github.com/ooni/probe-cli/releases

## Measurement improvements

We made a series of improvements to how OONI Probe measurements are
performed.

Specifically, we:

* Released a new version of Measurement Kit which includes DASH using
HTTPS:
https://github.com/measurement-kit/measurement-kit/releases/tag/v0.10.8
* Validated that the OONI Probe Telegram test works well by using jafar
(our censorship simulation tool) to trigger certain network conditions
and ensure that the data output is OK:
https://github.com/ooni/jafar/pull/15 &
https://github.com/ooni/jafar/pull/17
* Added support to probe-engine to use the orchestra to fetch the
configuration for Psiphon and to store locally the credentials for
logging-in to orchestra: https://github.com/ooni/probe-engine/pull/185

## OONI Probe 3.0 improvements

We added support for the clean shutdown of probe-cli, providing a clean
way of interrupting a running test on the OONI Probe desktop app:
https://github.com/ooni/probe-cli/issues/76

We also added support for filtering out incomplete test results
(https://github.com/ooni/probe-desktop/pull/92), and we upgraded the
styled-components to 4.0.0
(https://github.com/ooni/design-system/issues/49).

## OONI Explorer performance

As the new OONI Explorer fetches measurements from the OONI API,
improving the performance of OONI Explorer (in order to serve query
results faster and more reliably) required improving the performance of
the OONI API (see: https://github.com/ooni/api/issues/135).

We therefore did a lot of work on improving the OONI API. More
specifically, we:

* Added throttling to the OONI API in order to prevent abuse:
https://github.com/ooni/api/pull/139
* Removed SQLAlchemy models (resulting in performance improvements):
https://github.com/ooni/api/pull/153
* Refactored the list_measurements table (resulting in a huge
performance boost): https://github.com/ooni/api/pull/150

We also did several other important improvements to the OONI API and
data processing pipeline:

* Added support for detecting issues with the ingestion of data:
https://github.com/ooni/pipeline/issues/256
* Added support for anomalous and confirmed blocked measurement columns
in the fast-path pipeline: https://github.com/ooni/api/pull/141
* Added support for filtering by domain name for results in the
fast-path pipeline: https://github.com/ooni/api/pull/144
* Created two new tables for citizenlab/test-lists ingestion:
https://github.com/ooni/pipeline/pull/274
* Added tables that allow us to query by category code:
https://github.com/ooni/pipeline/issues/263
* Explored how to update one of our postgresql databases

## Infrastructure

Throughout December 2019, we spent a lot of time improving our
infrastructure and dealing with incidents.

Details around these incidents and what we did to resolve them are
available through the following tickets:
https://github.com/ooni/sysadmin/issues/410
https://github.com/ooni/sysadmin/issues/406

The infrastucture monitoring and alerting system that we have in place
enabled us to more easily identify these issues.

## Blog post to encourage LGBTQI site testing

In collaboration with OutRight Action International, we jointly
published a blog post to encourage OONI Probe testing of LGBTQI websites
around the world.

The blog post is published on the following sites:
   
* OONI: https://ooni.org/post/2019-measure-blocking-lgbtqi-sites/

* OutRight Action International:
https://outrightinternational.org/content/let%E2%80%99s-measure-blocking-lgbtqi-websites-around-world

VPNCompare published a blog post discussing our LGBTQI testing research:
https://www.vpncompare.co.uk/ooni-lgbtqi-censorship/

## Press coverage

An Intercept article was published to cover our joint research report
with Coding Rights & Women on Waves (titled "On the blocking of abortion
rights websites: Women on Waves & Women on Web"):
https://theintercept.com/2019/12/12/net-claro-e-vivo-bloqueiam-site-aborto-seguro/


## User-base

In December 2019, OONI Probe was run 9,270,932 times from 5,564
different vantage points in 209 countries around the world.

This information can also be found through our measurement stats on OONI
Explorer (chart on monthly coverage worldwide): http://explorer.ooni.org/

~ The OONI team.

-- 
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200110/0bb286d6/attachment-0001.sig>