[tor-project] OONI team status report April 2017

[Arturo Filastò]

Hello Tor!

Here are some updates related to what the OONI team has been up to in
April 2017.

# OONI Monthly Report: April 2017

The OONI team made steady progress in April 2017.

We released the stable version of ooniprobe's distribution for Raspberry
Pis ("Lepidopter"), as well as the beta version of our data processing
pipeline. We created proof of concepts for the orchestrator backend and
the client component, and we improved the monitoring of our
infrastructure. We also made progress towards new ooniprobe mobile and
desktop releases which feature improvements based on community feedback
and translations in 4 additional languages.

Below we provide some highlights from OONI's activities in April 2017.

## Stable release of ooniprobe's distribution for Raspberry Pis
("Lepidopter")

We released the stable version of ooniprobe's distribution for Raspberry
Pis (lepidopter). It includes the latest version of ooniprobe (2.2.0),
as well as a series of improvements and bugfixes.

The full changelog for this release can be found here:
https://github.com/TheTorProject/lepidopter/blob/master/ChangeLog.md#lepidopter-v100-2017-05-05

To install this version of the image follow the instructions detailed here:
https://ooni.torproject.org/install/lepidopter/

## Proof of Concept (PoC) for orchestrator backend

We have implemented a proof of concept prototype for the OONI
orchestrator backend specified here:
https://github.com/TheTorProject/ooni-spec/blob/master/opos/OONI-Probe-Orchestration-System-Design.md

In this iteration of the probe orchestration mechanism (source code
available here: https://github.com/TheTorProject/proteus).
The following functionality is supported:

* Ability for probes to register with the orchestration registry
(https://github.com/TheTorProject/proteus/tree/master/proteus-registry)
* Ability for probes to update the metadata about their probe
(https://github.com/TheTorProject/proteus/tree/master/proteus-registry)
* Support for sending push notifications via Apple Push Notifcation
service and Firebase Cloud Messagging
(https://github.com/TheTorProject/proteus/tree/master/proteus-notify)
* Support for administrators to schedule measurements via a web
interface
(https://github.com/TheTorProject/proteus/tree/master/proteus-frontend)
* Ability for probes to receive the tasks they have been notified about
and mark them as accepted, rejected or done
(https://github.com/TheTorProject/proteus/tree/master/proteus-events)
* Multiple architecture build system

The next steps include the deployment of the testing infrastructure for
the orchestration backend and starting to test it out inside of the
mobile apps.

## Proof of Concept (PoC) for orchestrator in probe

A proof of concept prototype was written to support orchestration inside
of measurement-kit:
https://github.com/measurement-kit/measurement-kit/pull/1210.

The client component supports the basic orchestration workflow that
includes:
   
* Probes registering with the orchestration system;
* Probes updating the probe metadata;
* Probe listing the tasks they should be running;
* Probes accepting, rejecting and concluding a task.

A sample command line interface that demonstrates the full orchestration
workflow has also been implemented to do end-to-end testing.

## New ooniprobe releases

We worked towards new releases of ooniprobe in April. The latest version
of ooniprobe includes:

* Integration of OONI's HTTP Header Field Manipulation test into
ooniprobe mobile apps

* Improvements to ooniprobe based on community feedback

* Translations in 4 additional languages: Arabic, Farsi, Russian and Greek

* Support for Right-To-Left languages in ooniprobe mobile and ooniprobe
desktop

ooniprobe now supports a total of 9 languages: English, Arabic, Farsi,
Russian, Hindi, Spanish, French, Italian and Greek.

## Beta version of next-generation pipeline

The pipeline is ready to process data on a daily basis, it has already
reduced our storage usage by factor of ~3x enabling faster data
processing. The pipeline still has some bugs to be investigated and some
features to be implemented to replace prev-generation pipeline, but all
stability bugs are fixed at the moment.

The beta version of the pipeline can be found at
https://github.com/TheTorProject/ooni-pipeline/pull/62

## Improvements to OONI infrastructure

We have done a series of improvements to the OONI infrastructure,
including adding support for better monitoring based on prometheus:
https://github.com/TheTorProject/ooni-sysadmin/pull/106 (and
https://github.com/TheTorProject/ooni-sysadmin/pull/108). Some probes
for sending us notifications when some of the core infrastructure
components (bouncer, collectors, test-helpers, ooni-explorer and
ooni-measurements) go down have been setup.

You can inspect the public stats by accessing the grafana frontend to
prometheus at the following URL (username: guest, password: guest):
http://prometheus.infra.ooni.io:3000/

## Outreach and community engagement

On 7th April we presented OONI at the International Journalism Festival
(IJF) in Perugia, Italy. Information about our talk, titled "Uncovering
Internet Censorship", can be found here:
http://www.journalismfestival.com/programme/2017/uncovering-internet-censorship-right-around-your-corner

We also hosted our monthly community meeting on
https://slack.openobservatory.org/ on 25th April (meeting minutes:
http://meetbot.debian.net/ooni/2017/ooni.2017-04-25-15.59.log.html). As
part of the meeting, we discussed the translation of ooniprobe and
approaches for upcoming data visualizations.

## Userbase

In April 2017 ooniprobe was run 57,471 times from 1,535 different
vantage points across 158 countries around the world.

This information can also be found through our stats here:
https://measurements.ooni.torproject.org/stats

~ The OONI team