[ooni-talk] OONI Monthly Report: June 2023

[Maria Xynou]

Hello,

This email shares OONI's monthly report for June 2023.

*# OONI Monthly Report: June 2023*

Throughout June 2023, the OONI team worked on the following sprints:

* Sprint 93 (5th - 18th June 2023)
* Sprint 94 (19th June - 2nd July 2023)

Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni

Highlights are shared in this report below.

*## Launch of OONI Explorer supporting localization*

We are thrilled to share that OONI Explorer (https://explorer.ooni.org/)
now supports localization! OONI Explorer is an open data resource on
internet censorship around the world.

As of June 2023, OONI Explorer is available in 8 new languages:
* Chinese: https://explorer.ooni.org/zh-cn
* Farsi: https://explorer.ooni.org/fa
* French: https://explorer.ooni.org/fr
* German: https://explorer.ooni.org/de
* Russian: https://explorer.ooni.org/ru
* Spanish: https://explorer.ooni.org/es
* Thai: https://explorer.ooni.org/th
* Turkish: https://explorer.ooni.org/tr

We thank the Localization Lab (https://www.localizationlab.org/) community
for making OONI Explorer – and real-time open data on internet censorship
worldwide – available in so many new languages!

*## OONI Probe Mobile*

We made some improvement to the code quality of OONI Probe Android by
making progress on de-duplicating the code used for running background
tests (https://github.com/ooni/probe-android/pull/572) and we made
improvements to some of the unit tests (
https://github.com/ooni/probe-android/pull/575). We also added the ability
to “select all”, “deselect all” category codes in the websites categories
to test settings (https://github.com/ooni/probe-android/pull/576).

*## OONI Probe user guides translated to German *

Thanks to a community member (Curtis Baltimore), the OONI Probe user guides
are now available in German!

The German user guides are available through the following links:
* OONI Probe Mobile User Guide:
https://ooni.org/de/support/ooni-probe-mobile
* OONI Probe Desktop User Guide:
https://ooni.org/de/support/ooni-probe-desktop

*## OONI Run*

We have started adding support to OONI Probe Android for parsing and
displaying OONI Run v2 descriptors (
https://github.com/ooni/probe-android/pull/577). We also made progress on
the web editor for OONI Run v2 links (https://github.com/ooni/run/pull/131)
and on the backend for adding support for storing and serving them, by
deploying it to the testing infrastructure (
https://github.com/ooni/backend/pull/678).

*## OONI Probe CLI*

We released OONI Probe CLI v3.17.4 (
https://github.com/ooni/probe-cli/releases/tag/v3.17.4). This release
updates to OpenSSL 1.1.1u (https://github.com/ooni/probe-cli/pull/1148),
 includes a test helper fix to correctly measure Twitter (
https://github.com/ooni/probe/issues/2488), and updates the GeoIP database
to 2023-06 to address a Facebook Messenger issue (
https://github.com/ooni/probe/issues/2485). We also released OONI Probe
v3.17.5 (https://github.com/ooni/probe-cli/releases/tag/v3.17.5),  which
updates to go1.19.10.

Moreover, we upgraded the main development branch to go1.20.5 (
https://github.com/ooni/probe/issues/2478) and we released OONI Probe CLI
v3.18.0-alpha (https://github.com/ooni/probe-cli/releases/tag/v3.18.0-alpha)
and v3.18.0 (https://github.com/ooni/probe-cli/releases/tag/v3.18.0).

In addition to the aforementioned patches that we also included in v3.17.x,
the v3.18.x series releases the following major features (please, refer to
the v3.18.0-alpha and v3.18.0 release notes for the complete changelog):

* dslx (
https://pkg.go.dev/github.com/ooni/probe-cli/v3@v3.18.0/internal/dslx): an
internal DSL for simplifying the process of writing and maintaining network
experiments;

* netemx (
https://pkg.go.dev/github.com/ooni/probe-cli/v3@v3.18.0/internal/netemx):
 support for writing integration testing using Gvisor (a TCP/IP stack in
userspace that allows us to simulate censorship and throttling in a
predictable environment).

We started working on dslx in December 2022 and we started working on
netemx in March 2023. We are excited to see these features finally landing
in a stable release!

While working on the v3.18.x release, we also updated the underlying
ooni/netem library (https://github.com/ooni/netem) used by the netemx
feature to emulate DNS spoofing (https://github.com/ooni/netem/pull/27).

By releasing v3.18.0 (https://github.com/ooni/probe-cli/releases/tag/v3.18.0),
which uses go1.20.5, we also addressed a community raised issue (
https://github.com/ooni/probe/issues/2478) where it was impossible to
compile OONI Probe using the go1.20.x compiler series.

*## Expanding OONI’s testing model to support richer testing input*

We continued to work on supporting richer testing input (
https://github.com/ooni/ooni.org/issues/1291). We tagged version v0.1.0 (
https://github.com/ooni/2023-05-richer-input/releases/tag/v0.1.0) of our
ooni/2023-05-richer-input prototype (
https://github.com/ooni/2023-05-richer-input). This release saves the stage
of the repository at the end of May 2023. In this initial prototype, we
implemented richer input as a list of flat “mini nettests” that an existing
nettest (e.g., Facebook Messenger) should execute to obtain a complete
measurement of all the relevant targets.

While this design was adequate for an initial prototype, in June 2023 we
spent additional time trying to engineer and simplify this design. In
particular, we investigated whether we could reuse the DSL introduced in
ooni/probe-cli v3.18.0-alpha (
https://github.com/ooni/probe-cli/releases/tag/v3.18.0-alpha) to express
richer inputs in a more flexible and composable way (
https://github.com/ooni/probe/issues/2494). This culminated in releasing
v0.2.0 (https://github.com/ooni/2023-05-richer-input/releases/tag/v0.2.0)
of the prototype, which included a possible richer input implementation
that was using a DSL instead of flat nettest targets.

A DSL-based design is more flexible because we can naturally compose
existing building blocks (e.g., DNS lookups, TCP connect, TLS handshake)
with other blocks whose task is that of calculating a nettest’s top level
keys (e.g., the `blocking` key of Web Connectivity).

We introduced a complete DSL for richer input and adapted Facebook
Messenger to use such a DSL (
https://github.com/ooni/2023-05-richer-input/pull/5,
https://github.com/ooni/2023-05-richer-input/pull/6).

*## Creating a throttling measurement methodology*

Regarding measuring throttling (https://github.com/ooni/ooni.org/issues/1296),
we focused on adding (to the DSL introduced in ooni/probe-cli v3.18.0-alpha
(https://github.com/ooni/probe-cli/releases/tag/v3.18.0-alpha)) prototype
support for collecting extra statistics during long downloads to Web
Connectivity LTE (the current experimental version of Web Connectivity) and
to the ooni/2023-05-richer-input (
https://github.com/ooni/2023-05-richer-input). Currently, the underlying
framework that we use for these three pieces of code collects up to 64
network events during and after the TLS or the QUIC handshake. While these
events naturally include the initial download of the body, we wanted to
extend this model to collect download speed snapshots.

To this end, we imported memoryless from m-lab/go (
https://github.com/ooni/probe-cli/pull/1163) to have support for sampling
at exponentially distributed (and truncated) intervals, which provides
PASTA properties (https://en.wikipedia.org/wiki/Arrival_theorem) when
observing network events. We then used this functionality to implement
collecting download speed snapshots (
https://github.com/ooni/probe-cli/pull/1166) and we added initial support
for measuring the download speed (
https://github.com/ooni/2023-05-richer-input/pull/4).

*## Creating a Social Media Censorship Alert System*

In May 2023, we continued to make progress towards building a Social Media
Censorship Alert System. Specifically, we added support for the
reprocessing of existing measurements, and we are currently running the
event detector on the backend host. As part of improvements to the overall
event detector logic, we fixed a bug affecting how columns are handled in
Pandas.

This work is documented through the following pull request:
https://github.com/ooni/backend/pull/651

*## Creating a Censorship Incident Reporting Platform*

We’re excited to share that we’re building a Censorship Incident Reporting
Platform!

New censorship events frequently emerge worldwide on an ongoing basis, but
producing an in-depth report and analysis on each event (in a timely
manner) is not easily scalable. Moreover, rapid response efforts benefit
from censorship findings that are easier and quicker to consume.

We therefore aim to create a Censorship Incident Reporting Platform which
will regularly provide short reports with summary information on emergent
censorship events based on OONI data and charts. This will enable
journalists and human rights defenders to more easily learn about and
respond to censorship events. Over time, the platform will become an
archive of known censorship incidents from around the world based on OONI
network measurement data.

In June 2023, we made significant progress towards creating a Censorship
Incident Reporting Platform. Specifically, we developed the backend API for
creating, updating and retrieving censorship incidents through OONI
Explorer (https://github.com/ooni/backend/pull/678). This will allow logged
in OONI Explorer users to create incidents that can then be retrieved by
any user on the OONI Explorer platform, allowing us to develop views for
them. The backend API feature was deployed on testing and production
infrastructure, allowing us to start working on the frontend for displaying
and editing the censorship incidents.

On the frontend part, we started working on the editor (
https://github.com/ooni/explorer/pull/862), which supports an enriched
markdown format that allows us to embed MAT charts within markdown
formatted text. It also allows us to define some metadata which can be used
for retrieving incidents matching a specific search criterion (e.g. all
censorship incidents pertaining to a particular country).

*## OONI backend*

In June 2023, we implemented the extraction of test helper metadata (URL
and type) in the fastpath (https://github.com/ooni/backend/pull/682), which
will enable us to track which test helper was used in Web Connectivity
measurements and to create dashboards to detect correlation between test
helper locations and test failures. We updated the test helper rotation
tool to deploy Vector and configure log forwarding on test helpers (
https://github.com/ooni/backend/pull/692), which will allow collecting logs
from the test helpers centrally and storing them. We also implemented an
initial form of alarming based on the test helper failure rate.

We added support for GeoIP lookup for the probes (
https://github.com/ooni/backend/pull/678), and we implemented an
experimental tool and dashboard to monitor the accessibility of our website
(ooni.org) through Tor. We created an experimental STUN server to perform
initial connectivity tests from countries that block many other protocols.

*## Automating censorship detection and characterization based on OONI
measurements*

In June 2023, we added support for storing IP metadata related to control
measurements (e.g. AS and country information) to be used as part of the
DNS analysis (https://github.com/ooni/data/pull/30). We also added support
for making it easier to perform migrations of the database SQL schema,
allowing us to more quickly enrich the database table with more features (
https://github.com/ooni/data/pull/30/commits/7c9b6a5c68841c86e553c2d60d5fbf48b6d66228
).

We also added support for correctly mapping unknown_failures (caused by
software bugs) to known failure strings, which allows us to consider more
kinds of measurements as part of the analysis (which would have otherwise
been considered as “failed”). This work is documented here:
https://github.com/ooni/data/pull/29

Finally, we added support for skipping the use of the dask parallel
processing engine, when no parallelism is needed, thereby increasing
performance (https://github.com/ooni/data/pull/31).

*## Interviewed Project Manager candidates*

In June 2023, we completed the first round of interviews for the OONI
Project Manager position (
https://ooni.org/post/2023-job-opening-ooni-project-manager/). Based on
these interviews, we created a shortlist for the second (and final) round
of interviews and coordinated with candidates. We also started the second
round of interviews with shortlisted candidates.

*## Blocking of social media in Senegal*

Following unrest erupted over the sentencing of opposition leader Ousmane
Sonko, OONI data from Senegal shows that ISPs started blocking access to
several social media platforms on 1st June 2023.

Specifically, OONI data shows the blocking of:
* WhatsApp:
https://explorer.ooni.org/chart/mat?probe_cc=SN&since=2023-05-03&until=2023-06-03&time_grain=day&axis_x=measurement_start_day&test_name=whatsapp
* Telegram:
https://explorer.ooni.org/chart/mat?probe_cc=SN&since=2023-05-03&until=2023-06-03&time_grain=day&axis_x=measurement_start_day&test_name=telegram
* Facebook, Instagram, YouTube, Twitter:
https://explorer.ooni.org/chart/mat?probe_cc=SN&since=2023-05-03&until=2023-06-03&time_grain=day&axis_x=measurement_start_day&axis_y=domain&test_name=web_connectivity

OONI data collected from Senegal shows that access to social media and
messaging platforms is blocked by means of TLS interference (i.e. TLS
handshakes timing out).

In response to the blocks, we shared relevant OONI data (and charts) on our
social media platforms (
https://twitter.com/OpenObservatory/status/1664611174854303745), with the
#KeepItOn advocacy community, and with Senegalese advocacy groups.

*## Blocking of F-Droid in China*

As of 22nd June 2023, OONI data collected from China shows the blocking of
F-Droid by means of DNS injection, where an unrelated IP address (e.g.
DropBox) is returned to the client (
https://explorer.ooni.org/m/20230622211826.539282_CN_webconnectivity_87bc828c0c6e5758).
This technique is commonly used in China to restrict access to services,
and OONI data shows that F-Droid is blocked on at least 4 networks in China.

OONI data on the blocking of F-Droid in China is available here:
https://explorer.ooni.org/chart/mat?probe_cc=CN&since=2023-06-13&until=2023-07-13&time_grain=day&axis_x=measurement_start_day&test_name=web_connectivity&input=https%3A%2F%2Ff-droid.org%2F

In response to the block, we shared OONI data (and charts) in both English
and Chinese on our social media platforms (
https://twitter.com/OpenObservatory/status/1673659531639894019). We also
reached out to F-Droid developers to share relevant data that could
potentially help with circumvention.



*## Community use of OONI data### Civil society response to the social
media blocks in Senegal*

As soon as the social media blocks emerged in Senegal on 1st June 2023,
Senegalese civil society groups ran OONI Probe and independently reported
on the blocks, sharing OONI data (
https://twitter.com/aliamsi/status/1664374471849902080). Similarly, other
internet freedom community members, such as Deutsche Welle’s Oliver Linow,
responded to the blocks by sharing OONI data (
https://twitter.com/OliverLinow/status/1664397344111042565).

*### Access Now statement in response to social media blocks in Senegal*

In response to the social media blocks in Senegal, Access Now published a
statement (citing OONI data):
https://www.accessnow.org/press-release/keep-people-connected-senegal/



*## Community activities### RightsCon 2023*

Between 5th-8th June 20213, OONI’s Elizaveta, Maria, and Arturo traveled to
Costa Rica to attend RightsCon 2023 (https://www.rightscon.org/).

As part of their participation, they presented OONI as part of the
following 5 sessions:

1) Session: “KeepItOn in 2022: successes, opportunities, struggles, and
lessons learned in the fight against election-related shutdowns” (hosted by
Access Now and KICTANET), 6th June 2023, OONI speaker: Elizaveta Yachmeneva

2) Session: “Data and Action: Using data analysis for incident response and
threat prevention” (hosted by the Center for Digital Resilience), 7th June
2023, OONI speaker: Arturo Filastò

3) Workshop: “Learn to investigate internet shutdowns through an Iran 2022
case study” (hosted by OONI and IODA), 7th June 2023, OONI speaker: Maria
Xynou

4) Session: “Internet blocking and the consequences for human rights?”
(hosted by Cloudflare), 8th June 2023, OONI speaker: Arturo Filastò

5) Lightning talk: “Investigating Internet shutdowns through open data”
(hosted by OONI, IODA, M-Lab), 8th June 2023, OONI speaker: Maria Xynou

We also participated in a private #KeepItOn meeting to discuss how telcos
can support the fight against internet shutdowns.

Overall, participating at RightsCon provided a great opportunity to
reconnect with many of our partners and community members in person, and to
learn from many of the sessions that we attended!

*### OONI workshop for journalists and civil society in Sierra Leone*

On 21st June 2023, OONI’s Elizaveta facilitated an online OONI workshop for
journalists and civil society organizations in Sierra Leone as part of the
hybrid Stakeholders Dialogue on Internet Shutdowns event organized by
Access Now and our Sierra Leone partner, CHRDI (
https://ooni.org/partners/chrdi/), ahead of Sierra Leone’s 2023 elections.

Information about the event is available through the following links:
https://twitter.com/chrdiorg/status/1671661565068623877
https://twitter.com/accessnow/status/1670860527541600256

The event (along with the OONI workshop) can be watched here:
https://www.youtube.com/watch?v=ho1wTgvIE-M

*### OONI webinar for civil society in Sierra Leone*

On 22nd June 2023, OONI’s Elizaveta presented OONI’s tools and methods to
civil society groups in Sierra Leone as part of Paradigm Initiative’s
“Defend, Connect, Report: Navigating Internet Censorship and Defending
Digital Rights in Sierra Leone’s Election” webinar in preparation for
Sierra Leone’s 2023 elections.

Information about the webinar is available here:
https://twitter.com/ParadigmHQ/status/1671091608840990722

*### OONI workshops at DSAConnectCon 2023*

Between 28th-29th June 2023, OONI’s Maria traveled to Johannesburg, South
Africa, to participate at DSAConnectCon, a 2-day community convening
organized by our partner, Digital Society of Africa (DSA). Learn about
DSA’s important work here: https://digitalsociety.africa/

As part of her participation, Maria co-facilitated 2 training sessions in
collaboration with IODA’s Amanda. These training sessions (“Investigating
Internet shutdowns through open data”) introduced participants to both OONI
and IODA tools, methods, and datasets, including hands-on exercises based
on real-world internet shutdown scenarios.

Maria also assisted participants with localizing OONI Probe into 7
additional African languages (Zulu, Ndebele, Nyanja, Swati, Mozambican
Portuguese, Tumbuka, Shona) during the Localization Lab’s (
https://www.localizationlab.org/) hands-on workshop on localizing internet
freedom tools.

*### OONI Community Meeting*

On 27th June 2023, we hosted the monthly OONI Community Meeting on our
Slack channel (https://slack.ooni.org/), during which we briefly discussed:

1) Social media blocks in Senegal.
2) F-Droid block in China.
3) Monitoring internet censorship in Sierra Leone during the country’s 2023
elections.
4) OONI Explorer being released in 8 new languages (encouraging further
localization efforts).

*## Measurement coverage*

In June 2023, 56,948,483 OONI Probe measurements were collected from 2,897
networks in 165 countries around the world.

This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/

~ OONI team.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/ooni-talk/attachments/20230712/3d1424d4/attachment-0001.htm>