[ooni-talk] OONI Monthly Report: June 2021

[Maria Xynou]

Hello,

Throughout June 2021, the OONI team worked on the following sprints:

* Sprint 40 - Manatee (1-6 June, 2021)
* Sprint 41 - Trumpetfish (7-20 June, 2021)
* Sprint 42 - Kujira (21-30 June, 2021)

Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni

Highlights are shared in this report below.

*## Building a web platform for test list updates*

We finalized the URL submission interface by implementing full UI and
backend support for submitting URLs from a logged in account. This will
allow community members who are not familiar with GitHub to contribute to
the Citizen Lab test lists (through a web platform).

Through the interface it is possible to propose the addition of new URLs,
edit the category codes or notes of existing URLs, and to propose the
deletion of existing URLs. When a set of changes is finalized, a pull
request is opened on behalf of the user on GitHub and reviewers of the
Citizen Lab test list repository (https://github.com/citizenlab/test-lists)
are able to merge them while using the existing review workflow (
https://github.com/ooni/url-prioritization/,
https://github.com/ooni/api/pull/252).

*## Building a web platform for URL prioritization*

We finalized the implementation of the URL prioritization frontend. This
allows members of the OONI team to define priorities for the testing of
certain URLs or category codes (from the Citizen Lab test lists) based on
OONI’s policy for URL prioritization (
https://ooni.org/post/ooni-smart-url-list-system/). This work is documented
here: https://github.com/ooni/url-prioritization/

*## Automatically confirming DNS-based censorship*

We are now in a position to automatically confirm instances of DNS-based
censorship (when blockpages are served)!

In June 2021, we added backend support for confirming DNS-based anomalies (
https://github.com/ooni/backend/issues/517), and we added new blockpage
fingerprints (both DNS and HTTP response body) to our database. This has
allowed us to automatically detect and confirm the blocking of many more
websites around the world. This work is tracked through the following
issue: https://github.com/ooni/backend/issues/518

OONI Explorer now displays many cases of automatically confirmed DNS-based
censorship from around the world. Examples include Malaysia (
https://explorer.ooni.org/search?until=2021-07-15&since=2021-06-15&probe_cc=MY&only=confirmed),
Thailand (
https://explorer.ooni.org/search?until=2021-07-15&since=2021-06-15&probe_cc=TH&only=confirmed),
Myanmar (
https://explorer.ooni.org/search?until=2021-07-15&since=2021-06-15&probe_cc=MM&only=confirmed),
and many other countries.

*## Improvements to the OONI Measurement Aggregation Toolkit*

We made progress on the development of the OONI Measurement Aggregation
Toolkit (MAT). Specifically, we added support for filling in missing values
inside of the charts so that multi-axis charts appear aligned properly and
we added a custom tooltip component to display the breakdown of anomalies,
confirmed and OK measurements in a given bar tick (
https://github.com/ooni/explorer/pull/600).

*## OONI backend*

We started reprocessing all OONI data using the fastpath. As part of this
work we also slightly changed the way in which files are formatted in s3
based on community feedback and needs. This work is tracked through the
relevant GitHub issue:  https://github.com/ooni/backend/issues/357

We made several improvements to how CORS headers are being added to the
OONI API responses. This work is needed in order to make the URL
submission, prioritization and login UIs work properly with the backend.
The relevant pull requests are the following:
https://github.com/ooni/api/pull/253 and
https://github.com/ooni/api/pull/255.

We developed a new tool for creating rotating test helpers. This allows us
to automate the process of spinning up new OONI Probe test helpers,
adjusting the needed DNS configuration, setting up TLS certificates for
them, and provisioning them with the required test helper code. This work
has been done as part of the following pull request:
https://github.com/ooni/pipeline/pull/357

We created a Debian package for the next generation Web Connectivity test
helper to reduce the ratio of false positives in measurements:
https://github.com/ooni/backend/issues/499 and
https://github.com/ooni/probe-cli/pull/374

*## Progress on browser-based OONI Probe*

We made progress in implementing the web-based OONI Probe. Specifically, we
came up with a new technique for measuring URLs that does not rely on the
favicon image loading trick, but rather relies on the new fetch API. By
configuring the fetch API to query using the “no-cors” mode, we are able to
use this as a signal to infer if a specific webpage is blocked or not (
https://github.com/ooni/probe-web/commit/50b83108866ab0b63cdf4bd3e0d05013c92836ca
).

*## Released OONI Probe iOS 3.0.0 with support for automated testing*

We released OONI Probe iOS 3.0.0 with support for automated testing:
https://github.com/ooni/probe-ios/releases/tag/v3.0.0

Similarly to the OONI Probe Android 3.0.0 release, the OONI Probe iOS 3.0.0
release enables users to have websites tested automatically every hour.
This release also includes the new Experimental card (with OONI’s new STUN
reachability test), as well as OONI backend proxy support.

*## Improvements to OONI Probe Mobile*

In response to community feedback, we added support for minimizing a
running test. This allows users to continue using OONI Probe while a test
is running (https://github.com/ooni/probe-android/pull/441). We also added
a badge that indicates (in the running test screen) that a backend proxy is
enabled when a test is running (
https://github.com/ooni/probe-android/pull/442).

In collaboration with Bloco, we added more tests to the OONI Probe Android
app. Specifically, we added tests for the data models:
https://github.com/ooni/probe-android/pull/438

*## Released OONI Probe Desktop 3.5.0 with RTL support*

We released OONI Probe Desktop 3.5.0 with RTL support:
https://github.com/ooni/probe-desktop/releases/tag/v3.5.0

OONI Probe Desktop now supports a number of languages that use
right-to-left (RTL) scripts, including Arabic and Farsi. We thank the
Localization Lab community for the translations (which can be reviewed on
Transifex: https://www.transifex.com/otf/ooniprobe/ ).

We also released OONI Probe Desktop 3.5.1 with UX improvements and 100%
translation support for Arabic and Farsi:
https://github.com/ooni/probe-desktop/releases/tag/v3.5.1

*## Improvements to OONI Probe Desktop*

We identified and fixed a critical bug in the auto-updater of OONI Probe
Desktop which resulted in updates not being carried out properly in
versions 3.5.0 and 3.5.1 (https://github.com/ooni/probe/issues/1699). The
fix has been shipped in OONI Probe Desktop 3.5.2:
https://github.com/ooni/probe-desktop/releases/tag/v3.5.2

We made improvements to how the header in the test lists is being
displayed. Specifically, the sticky header is now using a different
component which makes it appear less “jumpy” when scrolling through the
view: https://github.com/ooni/probe/issues/1683

*## New onboarding screen for crash reporting*

We added a new onboarding step to the OONI Probe mobile and desktop apps to
enable users to opt-in to submitting crash reports to us. We hope that this
will result in more users opting-in to submitting crash reports, enabling
us to have more crash logs to improve the stability of the OONI Probe
mobile and desktop apps. This work is documented through the following
ticket: https://github.com/ooni/probe/issues/1455

*## Developing a new OONI Probe test for the Snowflake pluggable transport*

We continued to make important progress towards developing and shipping a
new OONI Probe test for the Snowflake pluggable transport. In June 2021, we
added support for running the Snowflake pluggable transport inside of the
OONI Probe engine. This work is documented through the following ticket:
https://github.com/ooni/probe/issues/1565

*## Documentation on writing OONI Probe tests*

To encourage community contributions of new network measurement
experiments, we wrote a tutorial which explains how third party developers
can write OONI Probe tests. This is tracked through the following issue:
https://github.com/ooni/probe/issues/1689

As part of this documentation, we used the new OONI Probe test for the Tor
Snowflake pluggable transport as an example. The code of the tutorial is
real OONI Probe engine code, which means that it’s going to be tested and
run as part of our continuous integration process.

The entry point for the tutorial on writing OONI Probe tests can be found
here: https://github.com/ooni/probe-cli/tree/master/internal/tutorial

*## Test list updates*

In June 2021, we contributed updates to the following Citizen Lab test
lists:

* Cambodia: https://github.com/citizenlab/test-lists/pull/777,
https://github.com/citizenlab/test-lists/pull/782 (including URLs shared by
community members)
* Armenia: https://github.com/citizenlab/test-lists/pull/785 (including
URLs shared by CyberHub Armenia)
* Palestine and Israel: https://github.com/citizenlab/test-lists/pull/780

*## Research report on internet censorship in Azerbaijan*

In collaboration with Azerbaijan Internet Watch, we worked on writing,
editing, and finalizing our joint research report that examines internet
censorship in Azerbaijan over the last year. Based on our analysis, we
produced multiple charts (that communicate key findings), many of which
were included in the final report. We coordinated with Azerbaijan Internet
Watch on publishing the report in early July 2021.

*## New partnership with the Centre for Internet and Society (CIS), India*

We have been collaborating with India’s Centre for Internet and Society
(CIS) for more than a year, having previously collaborated on a joint
research publication investigating TLS blocking in India (
https://ooni.org/post/2020-tls-blocking-india/).

In early June 2021, we formalized our collaboration by (signing an MoU and)
establishing a partnership with the Centre for Internet and Society (CIS).
We are now proud to list them on the OONI Partners page (and feature their
work): https://ooni.org/partners/cis-india/

*## Planning the online OONI Partner Training events*

In preparation for the online OONI Partner Training events, we worked on
the following activities:

* Analyzed the feedback shared through the pre-training survey (
https://forms.gle/YU4TW62LrLKmCC7N9);
* Created (and shared) the final agenda of the 3-day OONI Partner Training
(based on feedback shared through the pre-training survey);
* Created (and shared) a Resources document (
https://ooni.org/documents/2021-ooni-partner-training-resources/ooni-resources.pdf)
that includes OONI-related resources, as well as other resources (on
censorship circumvention, digital security, monitoring internet
connectivity shutdowns, and other censorship measurement platforms) that
participants expressed interest in (as part of the pre-training survey
feedback);
* Created a post-training survey (https://forms.gle/n62ktvcfD1u7DgPMA) to
collect participant feedback after the training program and learn how we
can improve future events;
* Created an online form that participants can fill out to receive OONI
certificates for their participation in the OONI Partner Training;
* Created relevant slides, resources, and hands-on exercises for each of
the 6 training sessions of the OONI Partner Training;
* Practiced the tech demos that would be provided as part of the OONI
Partner Training to ensure that case studies that are relevant to
participants are covered;
* Logistical coordination with 112 invited participants (including those
from the July 2021 OONI Partner Training).

*## Hosted a 3-day online OONI Partner Training*

Between 28th-30th June 2021, we organized and hosted a 3-day online OONI
Partner Training for our partners from 16 countries in Africa and Latin
America (https://ooni.org/partners).

We also invited some of our partners who work internationally with research
and advocacy organizations, as well as circumvention tool projects (Access
Now, Internews, Committee to Protect Journalists, Internet Society,
Netalitica, Citizen Lab, Psiphon, Tor Project, and the Localization Lab).

Overall, 41 participants from 26 digital rights organizations attended this
3-day OONI Partner Training. To ensure a safe, friendly, and pleasant
experience for all during the training, we asked all participants to adhere
to OONI’s Code of Conduct (
https://github.com/ooni/ooni.org/blob/master/CODE_OF_CONDUCT.md) and to
follow the Chatham House Rule on Privacy (
https://www.chathamhouse.org/about-us/chatham-house-rule).

The goal of the OONI Partner Training was to share knowledge and skills on
OONI’s censorship measurement tools, dataset, and methodologies to enable
their use as part of research and advocacy efforts. We also aimed to
collect feedback on how to improve OONI tools to better serve community
needs.

The event was facilitated entirely online (over Zoom), and entailed two 1.5
hour sessions per day (over 3 days). We facilitated the following 6
sessions as part of the 3-day OONI Partner Training:

1) Information Controls Around the World

2) Introduction to Internet Censorship

3) Using the OONI Probe Apps

4) Using OONI Run to coordinate censorship testing

5) Using OONI Explorer to find censorship measurement data

6) Interpreting OONI data

Further information about the training sessions is available through our
relevant blog post: https://ooni.org/post/ooni-partner-training-2021/

At the end of the 3-day training, we shared a post-training survey (
https://forms.gle/n62ktvcfD1u7DgPMA) with all participants to collect their
feedback on how to improve future OONI training events.

To support ongoing learning, we shared a Resources document (
https://ooni.org/documents/2021-ooni-partner-training-resources/ooni-resources.pdf)
with participants that includes OONI-related resources, as well as other
resources (on censorship circumvention, digital security, monitoring
internet connectivity shutdowns, and other censorship measurement
platforms) that participants expressed interest in.

We also shared OONI training slides (
https://docs.google.com/presentation/d/1mpoB_ObjMH0EWQhyZRpKWnwl5PUHf-Sa5PKaVclDEH8/edit?usp=sharing)
that they can use (and customize) to facilitate OONI workshops for their
communities. We will be issuing OONI certificates for participation in the
OONI Partner Training 2021.

*## Google Summer of Code (GSoC) mentorship*

This summer, we have the pleasure to serve as the host organization of 2
Google Summer of Code (GSoC) students: Kathrin Elmenhorst and Shivam
Shekhar (https://twitter.com/OpenObservatory/status/1399761448826507264).

Starting from June 2021, Kathrin is working with us on developing OONI
Probe network experiments (
https://community.torproject.org/gsoc/ooni-probe-experiments/), while
Shivam is working with us on developing integration and unit tests for the
OONI Probe desktop app (
https://community.torproject.org/gsoc/ooni-probe-testing/). Throughout June
2021, we offered mentorship and reviewed the pull requests of GSoC students.



*## Community use of OONI data### Report on media censorship in Venezuela*

On 4th June 2021, our Venezuelan partner, VEsinFiltro (
https://ooni.org/partners/venezuela-inteligente/), published a research
report documenting the blocking of media websites in Venezuela. Their
report is available here:
https://vesinfiltro.com/noticias/2021-06-04_censura_digital/

As part of their research, they made use of OONI data (
https://twitter.com/vesinfiltro/status/1400976570160984068), along with
other measurement data.



*## Community activities### RightsCon 2021*

We attended RightsCon Online 2021, during which we had the opportunity to
speak in the following sessions:

1) Community Lab session: “Who are the perpetrators of internet shutdowns?
What we documented in 2020 and beyond” (organized by Access Now’s #KeepItOn
campaign), 7th June 2021

2) Strategy session: “Measuring internet disruptions during elections (and
other political events)” (organized by OONI and CAIDA), 7th June 2021

3) Tech demo: “Understanding internet censorship data through OONI
Explorer” (organized by OONI), 10th June 2021

4) Panel: “Confronting threats from all sides: building the anti-censorship
ecosystem” (organized by Internews), 10th June 2021

5) Community Lab session: “Prepare, prevent, resist: a guide for internet
shutdown advocacy strategy and resilience” (organized by Internews), 11th
June 2021

In collaboration with CAIDA, we facilitated a strategy session during which
we introduced participants to OONI and IODA’s tools, datasets, and
methodologies for investigating internet disruptions. We explained that
internet disruptions around the world tend to emerge in correlation with
political events, and we discussed strategies for monitoring potential
censorship events. As part of this session, we also had an opportunity to
collect community feedback for the improvement of our tools and methods.

We created a 10-minute OONI Explorer demo, which was uploaded on
RightsCon’s online platform. During RightsCon, we facilitated a live chat
during which participants brought their questions related to using OONI
Explorer.

We were invited by Internews to speak on their panel to discuss recent
censorship events in Myanmar, the challenges associated with measuring
throttling, and how community members can measure and respond to censorship
events worldwide. We also participated in Internews’ community lab session,
during which we briefly presented the resource we created on network
measurement tools for internet shutdown advocacy.

*### Paradigm Initiative’s Ayeta Digital Rights Toolkit*

Our Nigerian partner, Paradigm Initiative (
https://ooni.org/partners/paradigm-initiative/), published the Ayeta
Digital Rights Toolkit, which aims to address the growing need to
safeguard digital rights defenders, journalists, whistle blowers, and
others working with
sensitive information in the global South. The Ayeta Digital Rights Toolkit
is available here: https://paradigmhq.org/programs/digital-rights/ayeta/

OONI is featured as a recommended tool under the “Measuring Internet
Shutdowns and Censorship” section of the Ayeta Digital Rights Toolkit:
https://paradigmhq.org/wp-content/uploads/2021/04/Ayeta%20Toolkit%20-%20English%20Version.pdf

*### OONI outreach in Mali*

In mid-June 2021, a community member (Emmanuel Vitus) led conversations on
digital rights and digital security in Mali. As part of these
presentations, Emmanuel introduced civil society members in Mali to OONI (
https://twitter.com/emmavitus/status/1408410588770979842), as well as to
Paradigm Initiative’s Ayeta Digital Rights Toolkit and other measurement
tools.

*### Blocking of Twitter in Nigeria*

Starting from 4th June 2021, OONI data started to show the blocking of
twitter.com in Nigeria (and the block is ongoing):
https://explorer.ooni.org/search?until=2021-07-15&since=2021-06-15&domain=twitter.com&probe_cc=NG

In particular, OONI data shows that twitter.com is blocked on multiple AS
networks in Nigeria by means of IP blocking, while some ISPs appear to be
using some form of Deep Packet Inspection (DPI) on HTTPS connections (
https://twitter.com/OpenObservatory/status/1402674098644389891).

We analyzed relevant OONI measurements from Nigeria and shared our analysis
with our local partners. We also produced a chart that provides a per-ASN
breakdown of the blocking of twitter.com in Nigeria, which we shared with
Internet Society to include in their Pulse project (
https://pulse.internetsociety.org/shutdowns).

*### Spanish OONI Probe CLI guide*

A community member (Jacobo Nájera) wrote and published a guide in Spanish
which explains how you can install and use the OONI Probe Command Line
Interface (CLI):
https://www.jacobo.org/sondas-ooni-desde-la-linea-de-comandos/

*### OONI Community Meeting*

On 22nd June 2021, we hosted the monthly OONI Community Meeting on our
Slack channel (https://slack.ooni.org/), during which we discussed the
following topics:

1. Upcoming legacy probe deprecation in light of the discontinuation of v2
onion services by October 2021: Need for more community outreach and
awareness raising.

2. Are there laws against performing network measurements in China?
Discussion of potential risks associated with network measurement.

*## Userbase*

In June 2021, 11,459,826 OONI Probe measurements were collected from 5,060
AS networks in 200 countries around the world.

This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/

~ OONI team.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/ooni-talk/attachments/20210715/abe64602/attachment-0001.htm>