[ooni-talk] OONI Monthly Report: November 2021

[Maria Xynou]

Hello,

OONI's November 2021 status report is shared below.

*# OONI Monthly Report: November 2021*

Throughout November 2021, the OONI team worked on the following sprints:

* Sprint 51 (1st - 7th November 2021)
* Sprint 52 (8th - 21st November 2021)

Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni

Highlights are shared in this report below.

*## Published research report investigating internet shutdowns through
Mozilla telemetry*

In collaboration with IODA, we published a new research report which
analyzes Mozilla telemetry to investigate internet shutdowns in Myanmar,
Uganda, Belarus, and Iran over the last year.

Our research report is available here:
https://ooni.org/post/2021-investigating-internet-shutdowns-mozilla-telemetry/

Mozilla also published a blog post about our report:
https://blog.mozilla.org/data/2021/11/08/detecting-internet-outages-with-mozilla-telemetry-data/

The goal of our research was to evaluate whether Mozilla telemetry could be
a valuable resource for the Internet freedom community to investigate
Internet shutdowns. To this end, we analyzed an aggregated dataset of
network activity based on Firefox web browser usage worldwide (access to
which was provided to us by Mozilla).

To evaluate how useful Mozilla telemetry is for researching Internet
shutdowns, we:

* Selected case studies (known shutdown events from January 2020 onwards)
* Analyzed Mozilla telemetry
* Compared Mozilla telemetry with other public datasets

We checked whether Mozilla telemetry provides signals for:

* Myanmar: Internet outages following February 2021 military coup
* Uganda: Nationwide internet outage amid 2021 elections
* Belarus: Internet outages amid 2020 elections
* Iran: Internet outages following 2020 elections

Our detailed findings are documented in our report. Overall, our analysis
demonstrates that Mozilla telemetry is a valuable resource for
investigating Internet shutdowns worldwide, providing novel insights
through geographical data granularity.

We encourage Mozilla to publish aggregated Mozilla telemetry as open data
to support research and advocacy efforts investigating Internet shutdowns
worldwide. We also share further recommendations in our report:
https://ooni.org/post/2021-investigating-internet-shutdowns-mozilla-telemetry/#recommendations


*## Published blog post on Myanmar research paper findings*

A few months ago, the OONI team collaborated with researchers at IODA,
Kentik, UC San Diego, and University of Michigan / Censored Planet on a
research paper examining censorship events in Myanmar (through diverse
measurement datasets) following the military coup.

The paper, titled "A multi-perspective view of Internet censorship in
Myanmar", was published by FOCI 2021:
https://dl.acm.org/doi/pdf/10.1145/3473604.3474562

In November 2021, we published a blog post which summarizes the research
paper findings:
https://ooni.org/post/2021-multiperspective-view-internet-censorship-myanmar/


*## Blog post for Internet Society (ISOC)*

In November 2021, our partner, Internet Society (ISOC), published our guest
blog post where we discuss why collaboration and transparency is key to
internet measurement:
https://pulse.internetsociety.org/blog/internet-measurements-collaboration-and-transparency

This blog post was also cross-posted on the OONI blog:
https://ooni.org/post/2021-internet-measurements-collaboration-transparency/

*## OONI Probe Mobile*

We released OONI Probe Mobile 3.4.0 for Android (
https://github.com/ooni/probe-android/releases/tag/v3.4.0) and iOS (
https://github.com/ooni/probe-ios/releases/tag/v3.4.0).

This release includes the following new features and improvements:
* New label indicating when a VPN is being used
* Improved UI for requesting autorun permission (
https://github.com/ooni/probe/issues/1820)
* Logs are retained for 1 week
* Bug fix in RiseupVPN and Signal analysis logic

We also released OONI Probe Mobile 3.4.1 with further bug fixes and
improvements (https://github.com/ooni/probe-android/releases/tag/v3.4.1).

Following the release of OONI Probe Mobile 3.4.0, we started to observe an
increase in the volume of measurements contributed by automated runs. The
new label which indicates the use of a VPN will hopefully also help improve
OONI data quality.

We also worked on improvements to our build system for OONI Probe so that
it can be further automated through continuous integration. In particular,
we added support to build the Android and iOS dependencies as part of
continuous integration.

*### Interviewed mobile developer candidates*

In September 2021, we published a job opening to hire a new mobile
developer for OONI Probe (
https://ooni.org/post/2021-job-opening-ooni-mobile-developer/). We reviewed
applications and throughout November 2021, we interviewed candidates.

## OONI Run

Based on community feedback, we made improvements to the OONI Run (
https://run.ooni.io/) platform. Specifically, we rewrote the URL editor (
https://github.com/ooni/run/pull/74), we limited the OONI Run platform to
website testing (https://github.com/ooni/run/pull/79), and we deployed the
changes to production.

*## Integrating tor into OONI Probe*

We made progress on integrating tor into OONI Probe Mobile and OONI Probe
Desktop. In particular, we worked on adding support for building go-libtor
on Android, iOS, and Linux: https://github.com/ooni/probe/issues/1867

*## Expanding censorship measurement methodologies*

Two essential steps required for shipping our new Websteps experiment (for
measuring the blocking of websites) include:

* Comparing our new Websteps experiment with our current Web Connectivity
experiment;
* Deploying the new Websteps test helper in production.

In November 2021, we made considerable progress on both. To compare
Websteps with Web Connectivity, we developed test cases to measure accuracy
and to compare the two experiments in terms of bandwidth used, overall test
runtime, and accuracy at detecting blocking events (
https://github.com/ooni/probe/issues/1805). We also worked on polishing the
Websteps design documents and we shared them with community members who
have relevant expertise, in order to receive their feedback for the
improvement of our new experiment (https://github.com/ooni/probe/issues/1811).


We worked towards deploying and using the new Websteps test helper (
https://github.com/ooni/backend/issues/544). We started off with an
incremental rollout where only a few ASNs use the new test helper, and we
eventually plan to replace the old test helper.

We continued to investigate TLS parroting and interference (
https://github.com/ooni/probe/issues/1812), and we made several OONI Probe
CLI releases which include a series of improvements to our experiments
(improving the data quality of measurements), as documented through the
following releases:

https://github.com/ooni/probe-cli/releases/tag/v3.11.0-beta.2
https://github.com/ooni/probe-cli/releases/tag/v3.12.0-alpha.3
https://github.com/ooni/probe-cli/releases/tag/v3.12.0-alpha.2
https://github.com/ooni/probe-cli/releases/tag/v3.11.0-beta.1
https://github.com/ooni/probe-cli/releases/tag/v3.12.0-alpha.1

*## OONI Explorer*

We added support to the OONI Explorer Search page for filtering based on
failed measurements (https://github.com/ooni/explorer/pull/623). We also
worked towards making OONI Explorer easier to debug (to improve the quality
of OONI Explorer, and to make the MAT easier to debug) by improving how we
log and report errors in OONI Explorer.

*## OONI Measurement Aggregation Toolkit (MAT)*

We continued to make improvements to the OONI Measurement Aggregation
Toolkit (MAT). We measured the performance of the MAT UI components to
identify bottlenecks. We worked on improving the interactivity and
filtering of the table chart, we fixed the presentation of the horizontal
axis of the chart, we worked on improvements to one-dimensional charts, and
we worked towards making measurable performance improvements to the
two-dimensional (2-axis) chart pages with large datasets (
https://github.com/ooni/explorer/pull/622).

*## Building a web platform for test list updates*

We continued to make progress on the web platform (
https://test-lists.ooni.org/) that we’re building to enable community
contributions to the Citizen Lab test lists (
https://github.com/citizenlab/test-lists).

In particular, we made improvements to display the contributor and the date
of when a URL was last updated in the list view (
https://github.com/ooni/test-lists-ui/pull/15), we added a notifications
feature (https://github.com/ooni/test-lists-ui/pull/16) and a favicon (
https://github.com/ooni/test-lists-ui/pull/17), and we fixed a backend bug
affecting the platform’s submit button (
https://github.com/ooni/backend/issues/545).

*## OONI backend*

In November 2021, we started migrating our infrastructure over to
Clickhouse (an alternative database solution).

To this end, we reviewed the architecture by:
* Implementing Clickhouse in our fastpath data processing pipeline;
* Switching the fasthpath and API CI to bullseye;
* Implemented the experimental Measurement Aggregation Toolkit (MAT) using
Clickhouse, and added relevant tests;
* Implemented the experimental PG to the Clickhouse feeder;
* Updating the CI to test Clickhouse;
* Determining what to keep in PG (failure modes and recovery);
* Preparing an architecture diagram;
* Splitting the platform in tiers;
* Using raw SQL or SQLalchemy;
* Implementing an experimental PG to the Clickhouse feeder;
* Performing tests by using Clickhouse for the measurements API;
* Handling JSONL.

We also worked on the following backend activities:

* Reviewed the new monitoring host, and removed the old host;
* Handled unplanned and upcoming VM reboots;
* Investigated an issue affecting the timing of measurements (probe clock
issues);
* Fixed a bug resulting in probe failures (
https://github.com/ooni/probe/issues/1328);
* Started reviewing and cleaning up DNS records used by OONI;
* Tested switching some ASNs to the new test helper;
* Experimented with centralized logging management;
* Continued to monitor OONI measurement coverage from unattended runs on
OONI Probe Mobile and Desktop (particularly in light of the recent iThena
integration of OONI Probe).

*## Collaboration with Netalitica*

Netalitica researchers continued to do excellent work in reviewing and
updating the Citizen Lab test lists. In November 2021, we reviewed their
updates to the test lists for Nicaragua and Ecuador and shared feedback for
improvements.

*## Test list updates*

In November 2021, we contributed minor updates to the following Citizen Lab
test lists:

* Egypt: https://github.com/citizenlab/test-lists/pull/863
* Global: https://github.com/citizenlab/test-lists/pull/862

We also opened a pull request with updates to the test list for India (
https://github.com/citizenlab/test-lists/pull/864), contributed by
participants of the Bachchao Project’s 2-day test list event (
https://ooni.org/post/2021-event-india-test-list/).



*## Notable community use of OONI Probe and OONI data### OONI Probe
integrated into Murakami*

In November 2021, Measurement Lab (M-Lab) integrated OONI Probe into their
Murakami measurement tool (https://www.measurementlab.net/blog/murakami/).
This is documented through the following pull request:
https://github.com/m-lab/murakami/pull/103

We thank M-Lab for integrating OONI Probe into Murakami, as this will help
boost OONI measurement coverage!

*### VEsinFiltro report on censorship during 2021 regional elections in
Venezuela*

In November 2021, VEsinFiltro published a report documenting the blocking
of independent news media websites, political criticism sites, human rights
sites, multimedia dissemination sites, and circumvention tool sites during
the 2021 regional elections in Venezuela. Their report makes use of OONI
data, among other data sources.

VEsinFiltro’s report is available in:
* English: https://vesinfiltro.com/noticias/2021-election-blocks/
* Spanish: https://vesinfiltro.com/noticias/2021-bloqueos-elecciones/



*## Community activities### OONI presentation at ISOC Community Week 2021*

On 16th November 2021, OONI’s Maria presented OONI as part of the Internet
Society’s (ISOC) Community Week event.

Information about the ISOC Community Week 2021 is available here:
https://www.internetsociety.org/events/community-week/

*### OONI presentation at Tor’s State of the Onion 2021*

On 17th November 2021, OONI’s Arturo presented OONI as part of the Tor
Project’s annual State of the Onion 2021 event (
https://blog.torproject.org/state-of-the-onion-2021), during which he
shared OONI highlights from 2021, as well as upcoming OONI projects for
2022.

The event was live-streamed on YouTube:
https://www.youtube.com/watch?v=mNhIjtXuVzk

*## OONI training for civil society in the Gambia*

On 29th November 2021, OONI’s Maria facilitated an (online) OONI training
for human rights defenders in the Gambia in preparation for the country’s
2021 elections.

*### OONI Community Meeting*

On 30th November 2021, we hosted the monthly OONI Community Meeting on our
Slack channel (https://slack.ooni.org/), during which we discussed the
following topics:

1. OONI research reports published in November 2021.

2. Iranian government is using a new type of regional Internet censorship
during protests which blocks International internet access (except few
services) but allows local websites and services. It would be nice if an
OONI test could be developed to detect & log this scenario and submit the
result to OONI once the international connectivity is fully restored. This
is tracked through the following epic:
https://github.com/ooni/probe/issues/1600

*## Userbase*

In November 2021, 31,393,906 OONI Probe measurements were collected from
2,276 AS networks in 157 countries around the world.

This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/

~ OONI team.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/ooni-talk/attachments/20220114/a57c7792/attachment-0001.htm>