[tor-project] OONI Monthly Report: September 2017

Fri Sep 29 20:32:54 UTC 2017

# OONI Monthly Report: September 2017

The OONI team completed all deliverables for its current contract with the OTF.

## Stable release of Measurement API

We released the stable version of OONI's new measurement API, which is available here: https://api.ooni.io/.

The OONI API is based on the stable release of our data processing pipeline, which has been re-engineered over the last year to detect censorship events around the world faster and more accurately.

We added the following to the OONI API:

* API endpoints for listing and filtering anomalous measurements

* API endpoints for downloading full reports

We changed the following:

* Reverse sorting in `by_date` view and hide measurements from time travellers

* Better API documentation thanks to redoc based on OpenAPI 2.0

* Improve request validation thanks to connexion base on OpenAPI

* Oonify the UI

* Better testing

## Stable release of the OONI data processing pipeline

We now have a much stronger data processing pipeline (https://github.com/TheTorProject/ooni-pipeline/pull/62).

The scheduling of the pipeline is based on Airflow which is more robust than our prior luigi based solution. By doing this we have been able to reduce the number of silent outages going undetected for weeks.

The new version of the pipeline allows for much quicker feedback and interation, allowing us to run meaningful SQL queries that return result within minutes (not hours). This is also the foundation for building our new OONI measurements API.

We feel confident that the pipeline is going to easily scale to up to 100 GB of network measurement data per day. The current rate is of ~20GB of measurements per day and we are able to process this amount of data within an hour (~10 min for compression, ~10 min for sanitisation, ~30 min for inserting it into the DB), so we have some margin for scaling up to 100GB a day without significant modifcations to our infrastructure.
Currently the data is being processed in 24 hour batches, but based on the stats above, we are aiming to reduce that to 12h, 8h and eventually 1h.

The pipeline has been designed to be modular and allow us to easily ingest network measurement data from other projects too.

Currently the pipeline is analysing the data looking for blockpages and signs of DNS manipulation. We have plans for adding heuristics for detecting also more advanced insights in future versions.

For the future of the pipeline see: https://github.com/TheTorProject/ooni-pipeline/milestone/2

## Implement visualisations for the measurements interface

Elio Qoshi has started working for OONI as a designer. With him we have started to define the overall design and style of all OONI related content.

Some of this work has been integrated into our living style guide available at the following URL: http://openobservatory.github.io/design.

Part of the work also includes data visualisations that are used to visualise OONI data (see, for example, the BarChart component).

## Data visualisations for reports and blogs

Thanks to the work on the design front, we have started creating templates and guidelines for making charts that much such a task easier in the future. See: https://github.com/OpenObservatory/design/issues/2.

We created data visualizations based on our new style guide that are included in the following reports:

1. Cuba: https://ooni.torproject.org/post/cuba-internet-censorship-2017/#findings

2. Iran: https://ooni.torproject.org/post/iran-internet-censorship/#findings

## Published research report on internet censorship in Iran

In collaboration with ASL19, ARTICLE 19, and Small Media, we published a research report on internet censorship in Iran.

The report is available here: https://ooni.torproject.org/post/iran-internet-censorship/

We also published a summary of the report on the Tor blog: https://blog.torproject.org/internet-censorship-iran-findings-2014-2017

## Stable release of Orchestrator

We released the stable version of Probe Orchestration.

This features:

* Support for sending push notifactions to all mobile probes that include a message

* Support for having them run custom tests via the OONI Run custom URI scheme

* Better unittesting that also speak to the database to ensure everything is working as expected

See: https://github.com/TheTorProject/proteus/pull/28

For the time being we have decided to postpone including support for instrumenting probes to run measurements, following possible security concerns, see: https://github.com/TheTorProject/proteus/issues/24. Once we have come up with a more solid specification and a proof of concept implementation of this scheme, we would greatly appreciate feedback and maybe a third party security audit of our proposed approach.

## OONI Run

We released OONI Run, a custom URI scheme for OONI Probe mobile apps that enables users to:

* Engage others with testing the sites and running the OONI Probe tests (included in OONI mobile apps) of their choice

* Embed an OONI Probe widget to monitor the blocking of their site around the world

OONI Run is available here: https://run.ooni.io/

We also published a blog post that explains OONI Run: https://ooni.torproject.org/post/ooni-run/

## Community

We hosted our monthly community meeting on https://slack.openobservatory.org on 29th September 2017 at 14:00 UTC.

We discussed the following:

1. OONI Run: Questions and feedback

2. Announcement of OONI API release: Questions

## Userbase

In September 2017 ooniprobe was run 81,258 times from 1,595 different vantage points across 149 countries around the world.

This information can also be found through our stats here: https://api.ooni.io/stats

~ The OONI team

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20170929/265177c4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 208 bytes
Desc: Message signed with OpenPGP using AMPGpg
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20170929/265177c4/attachment-0001.sig>