[tor-project] OONI Monthly Report: July 2020

[Maria Xynou]

Hello,

Throughout July 2020, the OONI team worked on the following sprints:

* Sprint 16 - Neon (1st July 2020 - 5th July 2020)
* Sprint 17 - Θέτις (6th July 2020 - 19th July 2020)
* Sprint 18 - Colomber (20th July 2020 - 31st July 2020)

Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni

Highlights are shared in this report below.

## OONI Probe mobile app

In July 2020 we released:

* OONI Probe Mobile 2.5.1 on Android
(https://github.com/ooni/probe-android/releases/tag/v2.5.1) and iOS
(https://github.com/ooni/probe-ios/releases/tag/v2.5.1)

* OONI Probe Mobile 2.5.2 on Android
(https://github.com/ooni/probe-android/releases/tag/v2.5.2) and iOS
(https://github.com/ooni/probe-ios/releases/tag/v2.5.2)

The OONI Probe Mobile 2.5.1 release automatically deletes uploaded
measurements (creating more local storage space) and enables users to
stop a test while it’s running.

The OONI Probe Mobile 2.5.2 release ships the WhatsApp, Facebook
Messenger, and middlebox tests rewritten in golang.

## Adding support for configuring push notifications

As part of our ongoing efforts to add support for configuring push
notifications, we:

* Implemented countly on iOS: https://github.com/ooni/probe/issues/1209
* Implemented countly on Android: https://github.com/ooni/probe/issues/1208

## OONI Probe desktop app

Throughout July 2020, we continued to make progress on improving the
OONI Probe desktop app, as documented through the following pull requests:

https://github.com/ooni/probe-desktop/pull/164
https://github.com/ooni/probe-desktop/pull/163
https://github.com/ooni/probe-desktop/pull/162
https://github.com/ooni/probe-desktop/pull/161
https://github.com/ooni/probe-desktop/pull/159

We also created mockups for new settings to be included in the OONI
Probe desktop app. See: https://github.com/ooni/probe/issues/1225

## Making the OONI Probe apps rely entirely on the golang engine

We now have all OONI Probe tests written in golang!

As part of our work on making the OONI Probe apps rely entirely on the
golang engine, we:

* Rewrote the OONI WhatsApp experiment in Go:
https://github.com/ooni/probe-engine/issues/55
* Rewrote the OONI Facebook Messenger experiment in Go:
https://github.com/ooni/probe-engine/issues/736
* Removed the “All endpoints” option in the WhatsApp test:
https://github.com/ooni/probe/issues/1226
* Discussed how to improve the web interface of WhatsApp measurements:
https://github.com/ooni/probe-engine/issues/740
* Rewrote the OONI HTTP Invalid Request Line experiment in Go:
https://github.com/ooni/probe-engine/issues/761
* Rewrote the OONI HTTP Header Field Manipulation experiment in Go:
https://github.com/ooni/probe-engine/issues/763 
* Rewrote the OONI Web Connectivity experiment in Go:
https://github.com/ooni/probe-engine/issues/776
* Rewrote our SNI blocking experiment using urlgetter:
https://github.com/ooni/probe-engine/issues/684
* Updated to the latest version of probe-cli/probe-engine:
https://github.com/ooni/probe/issues/1178

As mentioned previously, we shipped WhatsApp, Facebook Messenger, and
middlebox tests rewritten in golang as part of the OONI Probe Mobile
2.5.2 release.

## Expanding OONI Probe measurement methodologies

As part of our ongoing efforts to improve and expand upon our
measurement methodologies, we enabled the measurement of private Tor
bridges and wrote an integration test, as documented through the
following tickets: https://github.com/ooni/probe-engine/issues/721 and
https://github.com/ooni/probe-engine/issues/629.

We also worked on investigating and remediating TLS false positives:
https://github.com/ooni/probe-engine/issues/769

## URL prioritization

We wrote an analysis script to add coverage metrics for URL
prioritization (https://github.com/ooni/backend/issues/425). This is
implemented in the following pull request:
https://github.com/ooni/pipeline/pull/319

This script will be used to monitor test list improvements
(https://github.com/ooni/backend/issues/418).

## Data analysis and pipeline work necessary for extracting per site metrics

We made progress on implementing API endpoints for returning top-level
measurement metadata: https://github.com/ooni/backend/issues/341

## Improving our server infrastructure

Throughout July 2020, we made important progress on improving the
monitoring of our infrastructure.

We created an analysis tool (“analysis runbook”) that runs a number of
threads to monitor the slow query summary and more. We also built a
grafana dashboard to receive alerts based on the monitoring of our
infrastructure. See: https://github.com/ooni/backend/wiki/Runbooks:Analysis

We also worked on setting up a new database host in Amsterdam:
https://github.com/ooni/backend/issues/400

## Testing and quality assurance

We implemented our own ASN database on the backend. This involved
investigations around data quality, automated generation and testing,
and deploying the ASN database in the pipeline and using it when
processing measurements. This work is tracked here:
https://github.com/ooni/backend/issues/406

We also:

* Worked on repairing a broken CI build in Measurement Kit:
https://github.com/measurement-kit/measurement-kit/issues/1926
* Improved upon oonimkall: https://github.com/ooni/probe-engine/issues/816
* Made sure we compiled a static miniooni for Linux so that it could be
run from any recent Linux distribution without worrying about the libc
ABI: https://github.com/ooni/probe-engine/issues/815
* Set up automatic, periodic CI runs:
https://github.com/ooni/probe-engine/issues/819
* Improved upon the OONI HTTP Header Field Manipulation test:
https://github.com/ooni/probe-engine/issues/821
* Wrote quality assurance tests for the OONI instant messaging and
middlebox experiments: https://github.com/ooni/probe-engine/issues/829
* Fixed a bug related to certain tests not running:
https://github.com/ooni/probe/issues/1233
* Fixed a bug in the OONI Telegram test:
https://github.com/ooni/probe-engine/issues/827
* Backported the Telegram test bugfix to a stable branch:
https://github.com/ooni/probe-engine/issues/828

Following a series of improvements, we released Measurement Kit
v0.10.12: https://github.com/ooni/probe-engine/issues/809

## Updated OONI Data Policy

On 22nd July 2020, we released the 1.4.0 version of the OONI Data Policy.

The updated OONI Data Policy is available here:
https://ooni.org/about/data-policy

The latest version (1.4.0) of the OONI Data Policy includes the
following changes:

* We now use our own ASN database (following the changes to MaxMind's
license).

*  You can no longer opt out of submitting country code information
(e.g. "IT" for Italy), as this significantly limits the value of
measurements (i.e. we have no idea where a test was run).

* We use a self-hosted analytics platform on our websites and OONI Probe
apps to better understand how our apps are used, improve our
understanding of user needs, improve OONI tools, and better serve the
OONI community. You can opt out of our use of analytics by disabling the
collection of app usage statistics in the OONI Probe app settings (and
you can opt out of our use of analytics on ooni.org and
explorer.ooni.org by unchecking the opt-out box at the end of our Data
Policy).

* In our Data Policy, we have added a section on the collection of app
usage statistics.

* In our Data Policy, we have added a section where we disclose what
information we will collect if you opt in to push notifications.

* OONI measurements are openly published by default, but we will not
publish data related to app analytics and push notification support
(which is securely stored separately from the public measurement metadb).

## Report on network performance in Northern Italy amid the COVID-19
pandemic

We collaborated with M-Lab on the publication of a research report that
shares measurement observations on network performance in Northern Italy
during the peak of the COVID-19 pandemic.

This report is available here:
https://ooni.org/post/2020-network-performance-covid19-italy/

In summary, we found:

* Significant increase in the number of NDT tests run per day around the
beginning of Italy’s nationwide lockdown (which started on 10th March 2020);

* Reduced NDT download and upload speed results during the nationwide
lockdown in Italy (between March 2020 to May 2020);

* Improved NDT download and upload speed results from early May 2020
onwards, when the Italian government gradually started to lift the
strict lockdown measures.

As the available measurements suggest that there may be a correlation
between increased internet use and reduced network performance, they
highlight the need to invest in internet infrastructure that is more
robust and sustainable in the long-term. They also highlight the need
for more longitudinal measurements collected from more stable vantage
points to improve our understanding of network performance issues.

OONI’s Simone also collaborated with M-Lab on a blog post that
introduces ndt7, which is available here:
https://www.measurementlab.net/blog/ndt7-introduction/#ndt7-introduction

## Internet Measurement Village 2020 recap blog post

Throughout June 2020, up until 3rd July 2020, we hosted the first
Internet Measurement Village (IMV), an online community event aimed at
sharing skills, knowledge, and resources on internet measurement.

As all the IMV sessions were live-streamed and will continue to live on
the OONI YouTube channel (https://www.youtube.com/c/OONIorg), we hope
that these recordings will serve as a valuable resource on internet
measurement for the internet freedom community.

In July 2020, we published a blog post to provide a recap of the
Internet Measurement Village 2020 and to share the slides and video
recordings of each presentation. This blog post is available here:
https://ooni.org/post/2020-imv-slides-recordings/

## Statement in support of the OTF

In July 2020, we published a second statement in support of the OTF amid
recent developments (our first statement is available here:
https://ooni.org/post/2020-06-19-save-internet-freedom-support-the-open-technology-fund/).

In our statement, titled “Support the OTF: Support a Free and Open
Internet”, we share our experience working with the OTF, and discuss how
they have played a crucial role in supporting the fight against internet
censorship worldwide.

This statement is available here: https://ooni.org/post/2020-support-otf/

## Created a new OONI partnerships page

We created a new page for the OONI website which features all of our
partners, shares their work, and the projects and/or research reports
that we have collaborated on.

This work is available here: https://github.com/ooni/ooni.org/pull/528

We will publish this page once it has been reviewed and approved by all
of our partners.

## Collaboration with Netalitica

As part of our ongoing collaboration with Netalitica, we continued to
review their test list updates.

In July 2020, we reviewed the updates to the Ethiopian, South African,
and Thai test lists and shared feedback. We also reviewed and shared
feedback on the Netalitica updates to the Algerian, Iraqi, Indonesian,
and Zimbabwean test lists.

## Data analysis

As part of our partnership with Azerbaijan Internet Watch, we analyzed
all OONI measurements collected from Azerbaijan between 1st January 2020
to 31st July 2020, based on which we produced relevant charts and wrote
a report.

We also performed data analysis for an OTF Information Controls Fellow
(https://github.com/ooni/ooni.org/issues/552) examining internet
censorship in Myanmar through the use of OONI Probe. Based on this
analysis, we produced a CSV file and several charts for the research fellow.

## Community activities

### NetGain Partnership webinar

On 8th July 2020, OONI’s Maria was invited to participate in the NetGain
Partnership online event, specifically in the webinar titled “Surging
Demand and the Global Internet Infrastructure”.

Information about the webinar is available here:
https://www.netgainpartnership.org/events/covid19/global-infrastructure

As part of her participation in this webinar, Maria presented OONI’s
work and discussed how increased internet use during the COVID-19
pandemic was correlated with decreased network performance (based on NDT
measurement analysis).

### RightsCon Online 2020

The OONI team attended RightsCon 2020, which took place entirely online
between 27th-31st July 2020. As part of our participation, we attended
many sessions, as well as the meetup of the KeepItOn campaign.

On 29th July 2020, OONI’s Maria co-facilitated the following session
along with Rama and Alberto from CAIDA: “Investigating internet
shutdowns with open data”.

As part of this session, Maria introduced participants to OONI for
measuring the blocking of websites and apps, while Rama and Alberto
introduced participants to IODA for measuring internet outages.

Maria was also on the RightsCon Online Program Committee, which involved
reviewing session proposals and providing advice for the “Network
Connectivity and Internet Shutdowns” track.

### Kill Switch Podcast

On 30th July 2020, OONI’s Maria was invited to speak on Access Now’s
Kill Switch podcast (https://www.volume.africa/kill-switch) that
discussed internet shutdowns around the world. As part of her
participation in the podcast, Maria discussed OONI and its role in the
KeepItOn campaign.

The podcast (Episode 4: “The Fight to Get Back Online”) is available
here:
https://player.captivate.fm/episode/61ad48c6-c67b-42ec-b8ca-9191915ec746

## Userbase

In July 2020, 7,567,361 OONI Probe measurements were collected from
5,765 networks in 209 countries around the world.

This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/

~ The OONI team.

-- 
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200827/2104e288/attachment.sig>