[ooni-talk] OONI Monthly Report: December 2022

[Maria Xynou]

Hello,

Below I share OONI's status report for December 2022.

*# OONI Monthly Report: December 2022*

Throughout December 2022, the OONI team worked on the following sprints:

* Sprint 79 (1st-4th December 2022)
* Sprint 80 (5th-18th December 2022)
* Sprint 81 (19th-31st December 2022)

Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni

Highlights are shared in this report below.

*## Year in Review: OONI in 2022*

We published our annual blog post, "Year in Review: OONI in 2022", which
shares:

* OONI highlights from 2022
* Highlights from the OONI community (including many of our partners)
* Our plans for 2023

You can read this post here: https://ooni.org/post/ooni-in-2022/

*## OONI’s 10th Anniversary*

On 5th December 2022, OONI celebrated its 10th anniversary! To mark this
milestone, we organized and hosted 2 live-streamed events, published a
timeline animation, community video, and blog post. Further details are
shared below.

*### Live-streamed events*

In celebration of OONI’s 10th anniversary, we organized and hosted 2
live-streamed events (https://ooni.org/post/10th-ooniversary-events/) on
our YouTube channel:

* 10th Ooniversary: OONI Highlights (5th December 2022)
* 10th Ooniversary: OONI Community (6th December 2022)

As part of the “OONI Highlights” live-streamed event, Arturo (OONI’s
founder) shared OONI’s history and highlights from the past 10 years, as
well as our plans for the future. This presentation can be viewed here:
https://www.youtube.com/watch?v=f4D4gq9TBMg

As part of the “OONI Community” live-streamed event, we invited 5 community
members who shared how they’ve used OONI tools and data as part of their
research and advocacy efforts (
https://ooni.org/post/10th-ooniversary-events/#10th-ooniversary-ooni-community).
Their presentations can be viewed here:
https://www.youtube.com/watch?v=Co_f8KPBIZM

Both events included presentations followed by discussion addressing
questions shared by participants through the live chat and shared pads.

On 5th December 2022, we also hosted an online Distance Disco (
https://distancedisco.nl/dd/ooni-s-10th-birthday-party) to dance with
community members in celebration of OONI’s 10th anniversary.

*### New animation of OONI highlights*

On 5th December 2022, we published an animation that provides a timeline of
OONI highlights from the past 10 years. This animation was produced in
collaboration with Robotina (https://www.robotina.it/).

The animation is available here: https://www.youtube.com/watch?v=AjpfF2kQ9jA

*### New OONI Community Video*

In celebration of OONI’s 10th Anniversary, we published a new video with
OONI community members.

You can watch this video here: https://www.youtube.com/watch?v=zrclQ2QZjVo

The video features 11 community members discussing how OONI has been useful
to their work, and what they would like to see OONI do in the future.

You can learn about the featured community members through our blog post:
https://ooni.org/post/ooni-community-video/

*### Highlights: 10 Years of OONI*

To mark OONI’s 10-year anniversary, we published a blog post where we
document the key OONI highlights from the past 10 years, as well as some of
our future plans.

This blog post is available here:
https://ooni.org/post/highlights-10-years-of-ooni/

*## OONI Probe Mobile*

We released OONI Probe Android 3.7.3 (
https://github.com/ooni/probe-android/releases/tag/v3.7.3) and OONI Probe
iOS 3.73 (https://apps.apple.com/us/app/id1199566366) with bug fixes and
improvements. We worked towards ensuring that OONI Probe Mobile uses the
check-in API instead of the test list API (
https://github.com/ooni/probe/issues/2390), we fixed broken Android tests (
https://github.com/ooni/probe/issues/2386), and we worked towards ensuring
that the testing of OONI Run links does not stop if a device is locked (
https://github.com/ooni/probe/issues/2361).

*## Published a new OONI Run user guide*

To enable community members to use the OONI Run (https://run.ooni.io/)
platform for custom website testing (and for coordinating censorship
measurement campaigns around the world), we created and published a new
user guide for OONI Run.

Our OONI Run user guide can be accessed here:
https://ooni.org/support/ooni-run/

Through the OONI Run user guide, you can learn how to:

* Create OONI Run links for testing websites (of your choice) for
censorship;
* Use OONI Run links with the OONI Probe mobile app;
* Share OONI Run links with other OONI Probe mobile app users;
* Generate OONI Run widget code.

Upon reading this guide, we hope you will feel empowered to share your OONI
Run knowledge and skills with others! We also hope this guide helps with
coordinating OONI Probe website testing around the world.

*## OONI Probe CLI*

We released OONI Probe CLI 3.16.6 (
https://github.com/ooni/probe-cli/releases/tag/v3.16.6) and OONI Probe
3.16.7 (https://github.com/ooni/probe-cli/releases/tag/v3.16.7). These two
releases include improvements to the experimental Web Connectivity v0.5
nettest, as well as the following important fixes:

* In OONI Probe CLI 3.16.6, we included the reliability fix (developed in
November) that makes OONI Probe try all the available test helpers, rather
than just the first one in the list;

* In OONI Probe CLI 3.16.7, we backported fixes for the WhatsApp and
Telegram experiments that stop measuring unencrypted web endpoints to avoid
possible false positives (an issue discussed in detail here:
https://github.com/ooni/ooni.org/issues/1317).

We continued the process of merging ooniprobe (the official CLI client) and
miniooni (the experimental CLI client) by introducing a common database
abstraction that could be used by both in the following pull request:
https://github.com/ooni/probe-cli/pull/988

*## OONI Probe Desktop*

We released OONI Probe Desktop 3.8.3 to include fixes from OONI Probe CLI
3.16.7 https://github.com/ooni/probe-desktop/releases/tag/v3.8.3

*## Expanding OONI’s testing model to support richer testing input*

To improve the quality of OONI measurements (and reduce the risk of false
positives), we stopped measuring unencrypted web endpoints for WhatsApp and
Telegram (https://github.com/ooni/ooni.org/issues/1317). We continued our
refactoring work to support providing richer testing input (
https://github.com/ooni/ooni.org/issues/1291,
https://github.com/ooni/ooni.org/issues/1292,
https://github.com/ooni/probe/issues/2362).

During December 2022, we focused on cleaning up “probeservices” (the Go
package that communicates with the OONI API) to set the stage for
implementing richer input. We removed support for deprecated APIs (
https://github.com/ooni/probe/issues/2380), we started to sync up the data
format with the backend (https://github.com/ooni/probe-cli/pull/996), and
we added support in the backend and in ooniprobe for receiving compressed
responses (https://github.com/ooni/probe/issues/2379). The latter change,
in particular, is required because adding more testing options to each URL
significantly increases the size of the response message. We then annotated
code to communicate with the backend with the request and response message
types (https://github.com/ooni/probe-cli/pull/1010,
https://github.com/ooni/probe-cli/pull/1011). This change allows us to
generate a swagger describing ooniprobe’s view of the API, which we can
compare to the one generated by the server, to be sure the two
implementations are (and stay) in sync.

Regarding richer input, we started sketching out a design document (
https://github.com/ooni/probe/issues/2381) and a prototype (
https://github.com/ooni/probe-cli/pull/1005) implementing this design
document. The general idea is to give experiments freedom to call the
specific backend API they require for getting their inputs, which removes
the need to maintain middleware code that delivers inputs to the
experiments. With more input diversity, such a middleware code becomes
increasingly the bottleneck to innovation, so we want to experiment with
removing it and adding additional functions that experiments can call to
implement common functionality instead.

We also added support to the Web Connectivity Test Helper (TH) for
measuring HTTP/3 hosts (https://github.com/ooni/probe-cli/pull/985).
Support for HTTP/3 in the TH is crucial to start measuring websites using
HTTP/3 in the Web Connectivity experiment.

Reflecting on our own experience writing experiments in the step-by-step
style (
https://github.com/ooni/probe-cli/blob/master/docs/design/dd-003-step-by-step.md),
as well as by learning from users’ contributions (
https://github.com/ooni/probe-cli/pull/989#discussion_r1032544878), we
determined that the step-by-step API provides the basic building blocks for
writing all the experiments we need, but there is also a need to have more
user-friendly and intuitive wrappers on top of it. To this end, we started
experimenting with a simple domain specific language for expressing and
composing low-level operations that should simplify the writing of
experiments (https://github.com/bassosimone/oonidsl/).

*## OONI Explorer*

We started working towards integrating relevant third-party data into OONI
Explorer. Specifically, we worked on third-party data visualizations on
country and network pages using the internet outage data from IODA and
internet traffic data from Cloudflare. We also continued conducting UX
research to improve the OONI Explorer measurement pages (
https://6q8h6s64.optimalworkshop.com/optimalsort/avnrhnv5/sort).

We added support for choosing the time granularity on the Measurement
Aggregation Toolkit (MAT), beyond day granularity (which has been the only
option so far). Users can now also display the data per hour, week and
month (https://github.com/ooni/explorer/pull/818).

*## Published an OONI Explorer demo*

To enable community members to use OONI Explorer (https://explorer.ooni.org/)
for investigating internet censorship based on OONI data, we published an
OONI Explorer demo on our YouTube channel.

The OONI Explorer demo is available here:
https://www.youtube.com/watch?v=6Rce-xshLac

*## OONI backend*

In response to a change in the testing of WhatsApp that made our tests
fail, we implemented a temporary workaround that restores the functionality
of the test (https://github.com/ooni/pipeline/pull/403).

We added support for HTTPS to the deb.ooni.org Debian package repository
after it was requested by the community to circumvent blocking of the HTTP
version. We also moved the service to the new backend host.

We enabled gzip support in Nginx to compress responses from the check-in
API and other JSON content and created a simple dashboard to monitor the
improvement. This makes the responses from the Measurement Aggregation
Toolkit (MAT) quicker for the browser to download.

*## Automating censorship detection and characterization based on OONI
measurements*

As part of our ongoing effort to automate and improve our characterization
of censorship events through the analysis of OONI measurements, we made
significant progress on the ooni/data tool (https://github.com/ooni/data).

Following internal technical discussions, as well as conversations with
members of the OONI community, we implemented a new iteration of the data
model used for representing the results of experiments (
https://github.com/ooni/data/pull/19).

Moreover, we started the process of evaluating the new analysis engine
against the analysis that currently exists and is mostly performed as part
of the measurement workflow (https://github.com/ooni/data/issues/21). Some
very preliminary findings suggest that it’s quite an important improvement
both in terms of accuracy and recall.

We also experimented with producing plots that breakdown the means through
which blocking is implemented (https://github.com/ooni/data/issues/22).

Smaller incremental improvements include:
Handling the deduplication of already processed data to support the easier
reprocessing of data (
https://github.com/ooni/data/pull/19/commits/eea26f7138b14fc4595d55b432d287bb96f2fa6a);

Expanding cloud provider detection to reduce DNS false positives (
https://github.com/ooni/data/pull/19/commits/01dc691d695dfc103d604846ad7568423d77375a);

Refactoring of the DNS anomaly analysis subsystem (
https://github.com/ooni/data/pull/20).

We also started implementing the extraction of the blocking type in the
fastpath pipeline.

*## Creating a Social Media Censorship Alert System*

We continued to work on the development of the Social Media Blocking Event
detector. We started running a prototypal version, created a dashboard and
a jupyter notebook. We ran the prototype for weeks to review the initial
results and improve the algorithm over time.

*## Published Test Lists Editor screencast*

To enable community members to contribute websites for censorship testing,
we created and published a new screencast for our Test Lists Editor (
https://test-lists.ooni.org/).

The Test Lists Editor screencast can be viewed here:
https://www.youtube.com/watch?v=6i2OVHUQEpE

Through our Test Lists Editor, you can review and contribute websites for
censorship testing by OONI Probe users around the world.

*## TikTok blocked in Jordan*

On 16th December 2022, Jordan blocked access to TikTok amid fuel protests.
We rapidly responded by sharing relevant OONI data, charts, and information
about the block on Twitter:
https://twitter.com/OpenObservatory/status/1603845292343889924

*## Test list updates*

Following the blocking of TikTok in Jordan, we added TikTok endpoints to
the test list for Jordan (https://github.com/citizenlab/test-lists/pull/1201).
Thanks to our URL prioritization system, these newly added URLs were
immediately prioritized for testing in Jordan as soon as the pull request
was merged (enabling the immediate collection of relevant measurements and
supporting rapid response efforts).

Throughout December 2022, we also reviewed and merged many other updates to
test lists contributed by community members:
https://github.com/citizenlab/test-lists/pulls?q=is%3Apr+is%3Aclosed

*## Creating a new OONI Outreach Kit*

Based on review and feedback, we continued to make improvements to the
materials and resources of the upcoming OONI Outreach Kit. Specifically, we
made extensive edits and improvements to the set of OONI workshop slides
included in the OONI Outreach Kit.

*## Updated OONI Partner pages*

On our website, we have an “OONI Partners” section which features our
partners, their important work, and the specific projects and research
reports that we’ve collaborated on (https://ooni.org/partners).

We updated many OONI Partner pages to include our latest collaborations:
https://github.com/ooni/ooni.org/pull/1328

*## Cloudflare blog post *

Cloudflare published a blog post describing how civil society organizations
use Cloudflare Radar (https://radar.cloudflare.com/) to track internet
shutdowns around the world:
https://blog.cloudflare.com/partnering-with-civil-society-to-track-shutdowns/

This blog post highlights OONI, where we share how Cloudflare Radar has
been useful to our work.



*## Community use of OONI data### iMAP 2022: Research reports on internet
censorship in 8 Asian countries*

On 21st December 2022, our long-term partner, Sinar Project, published 8
new research reports on internet censorship in Southeast Asia and Hong Kong
(China) in collaboration with their Internet Monitoring and Action Project
(iMAP) partners (https://imap.sinarproject.org/). Based on the analysis of
OONI data, these research reports investigate internet censorship in
Malaysia, Thailand, Myanmar, Indonesia, Cambodia, Vietnam, Hong Kong, and
the Philippines.

Their research reports can be found here:
https://imap.sinarproject.org/reports/2022

Our iMAP partners launched and presented these 8 research reports through
the following live-streamed presentation:
https://www.youtube.com/watch?v=u9JsCqXebPI

To highlight and help disseminate their important work, we published a blog
post which briefly discusses and links to their 8 research reports:
https://ooni.org/post/2022-imap-8-research-reports-southeast-asia/



*## Community activities### Internet Governance Forum (IGF) 2022*

On 1st December 2022, OONI’s Maria presented OONI on the panel “Help! The
Kill switch is taking away my limited agency” at the Internet Governance
Forum (IGF) 2022 (
https://intgovforum.org/en/content/igf-2022-town-hall-54-help-the-kill-switch-is-taking-away-my-limited-agency).


*### OONI workshop for journalists in Kyrgyzstan*

On 20th December 2022, OONI’s Elizaveta facilitated an online OONI workshop
for journalists in Kyrgyzstan as part of a training organized by Internews.
During this workshop, Elizaveta explained how journalists can use OONI
Probe and OONI data to investigate internet censorship.

*## Userbase*

In December 2022, 57,009,977 OONI Probe measurements were collected from
2,815 AS networks in 161 countries around the world.

This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/

~ OONI team.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/ooni-talk/attachments/20230111/0492c0af/attachment-0001.htm>