[tor-project] OONI Monthly Report: November 2020
Maria Xynou
maria at openobservatory.org
Wed Dec 9 17:40:59 UTC 2020
Hello,
Throughout November 2020, the OONI team worked on the following sprints:
* Sprint 25 - Näkki (October 26 - November 8, 2020)
* Sprint 26 - Neerali (November 9 - 22, 2020)
* Sprint 27 - Hexacorallia (Nov 23 - Dec 6, 2020)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Highlights are shared in this report below.
## OONI Probe Mobile
### Released OONI Probe Mobile 2.7.1
We released OONI Probe Mobile 2.7.1 for:
* Android: https://github.com/ooni/probe-android/releases/tag/v2.7.1
* iOS: https://github.com/ooni/probe-ios/releases/tag/v2.7.1
This release includes the following noteworthy changes:
* We have removed the "Include Network Info" setting (which enabled you
to opt-out of ASN collection). Now ASN information
(https://ooni.org/support/glossary/#asn) is collected by default. We
decided to do this because measurements that don't include ASN
information are non-actionable.
* We have removed the setting which enabled you to share your IP address
(on an opt-in basis). We decided to remove this setting because we think
the potential risk associated with sharing IP addresses is greater than
the benefit. Furthermore, when users did choose to share their IP
addresses over the last years, we did not find them particularly useful.
Following these changes, the "Sharing" setting is no longer available in
the OONI Probe mobile app (since the above options have been removed),
and the "Publish Results" setting (which is enabled by default) has been
moved under the Privacy tab of the app.
Through these changes (in OONI Probe Mobile 2.7.1), we aim to make OONI
Probe measurements both more secure and more actionable.
### Progress on adding support for automated regular testing
We continued to make progress towards adding support in the OONI Probe
mobile app for automated regular testing, as documented through this
ticket: https://github.com/ooni/probe/issues/916
We also worked on activities pertaining to the following tickets:
https://github.com/ooni/probe/issues/1269
https://github.com/ooni/probe/issues/1271
https://github.com/ooni/probe/issues/970
https://github.com/ooni/probe/issues/1249
In addition, we released the beta version of OONI Probe Android 2.8.0:
https://github.com/ooni/probe-android/releases/tag/v2.8.0-beta.1
## Updated OONI Data Policy
We updated the OONI Data Policy to reflect the changes made in the OONI
Probe Mobile 2.7.1 release. The updated version (1.4.1) of the OONI Data
Policy is available here: https://ooni.org/about/data-policy
This is a minor update, mainly aiming to reflect that OONI Probe mobile
app users can no longer opt-out from ASN collection, nor opt-in to IP
address collection. These options are currently available through the
OONI Probe desktop app (though future releases will remove these options
too).
## OONI Probe Desktop
### Released OONI Probe Desktop 3.1.0
We released OONI Probe Desktop 3.1.0 for macOS and Windows:
https://github.com/ooni/probe-desktop/releases/tag/v3.1.0
With the latest version (3.1.0) you can:
* Choose the categories of websites that you want to test (by
enabling/disabling website categories in the settings)
* Change the language of your OONI Probe desktop app (thanks to support
from the Localization Lab community!)
The latest version also features a new settings screen layout.
In preparation for the OONI Probe Desktop 3.1.0 release, the
Localization Lab coordinated the translation of new OONI Probe strings.
We also translated the new strings to Greek ourselves.
Towards the end of November 2020, we released OONI Probe Desktop 3.1.1
with some bug fixes:
https://github.com/ooni/probe-desktop/releases/tag/v3.1.1
We also worked on the activities documented through the following
tickets and pull requests:
https://github.com/ooni/probe/issues/1243
https://github.com/ooni/probe-desktop/pull/185
https://github.com/ooni/probe/issues/1274
https://github.com/ooni/probe/issues/1290
https://github.com/ooni/probe/issues/1291
### Updated OONI Probe macOS homebrew package
We worked on updating the OONI Probe macOS homebrew package with the
OONI Probe Command Line Interface (CLI).
This work is documented through the following tickets:
https://github.com/ooni/probe/issues/1270
https://github.com/ooni/probe/issues/1289
We have opened a pull request for the new OONI Probe homebrew package
here: https://github.com/Homebrew/homebrew-core/pull/66474
## OONI Probe Command Line Interface (CLI)
We continued to improve upon the OONI Probe CLI through the following
releases:
* 3.0.10: https://github.com/ooni/probe-cli/releases/tag/v3.0.10
* 3.0.11: https://github.com/ooni/probe-cli/releases/tag/v3.0.11
## Building Debian package for OONI Probe
We made progress on building a Debian package for OONI Probe. Our work
on this is available here: https://github.com/ooni/probe-cli/pull/162
The Debian package for OONI Probe could optionally be periodically
invoked by systemd to perform automatic testing.
## OONI Probe Engine
In November 2020, we made the following improvements to the OONI Probe
measurement engine:
* Implemented fallback for lookup IP:
https://github.com/ooni/probe-engine/issues/1026
* Routine releases: https://github.com/ooni/probe-engine/issues/1004
* Removed the ASN, country code, and IP settings (in line with the OONI
Probe Mobile 2.7.1 release):
https://github.com/ooni/probe-engine/issues/974
* Reverted the sanitization of the resolver IP in the Web Connectivity
test: https://github.com/ooni/probe-engine/issues/1047
* Bug fix: https://github.com/ooni/probe-engine/issues/994
## Expanding OONI Probe measurement methodologies
As part of our ongoing efforts to expand our measurement capabilities, we:
* Wrote the Web Connectivity test helper in Go:
https://github.com/ooni/probe-engine/pull/1078
* Split TLS ClientHello to better characterise SNI blocking (as part of
research on how to measure TLS middleboxes):
https://github.com/ooni/probe-engine/issues/622
* Added support to the http3 experiment (contributed by community
members) for using an alternative resolver:
https://github.com/ooni/probe-engine/issues/1024
* Wrote a basic specification for the `urlgetter` experiment:
https://github.com/ooni/spec/pull/205
* Made `urlgetter` experiment measurements public through the OONI API:
https://github.com/ooni/probe-engine/issues/1056
We also fixed a bug which occurred when the new RiseupVPN experiment
(contributed by community members) ran after the OONI Psiphon
experiment, resulting in the failure of the RiseupVPN experiment. This
bug occurred because the RiseupVPN experiment was mistakenly forced to
pass through the tunnel created by Psiphon. This was caused by an
implementation detail where the tunnel created by Psiphon was associated
with the measurement session and subsequently reused by other
experiments (in this case, the RiseupVPN experiment). The fix therefore
entailed changing the codebase so that the OONI Psiphon experiment uses
its own private tunnel and does not share it with the measurement
session. This work is documented through the following ticket:
https://github.com/ooni/probe-engine/issues/1050
## OONI Explorer
### Website-centric stats page
We made progress on creating a website-centric stats page for OONI
Explorer: https://github.com/ooni/explorer/pull/475. The goal of this
page is to present users with OONI measurement stats and visualizations
based on a tested website around the world.
We created an initial version of this page, which is available here:
https://explorer.ooni.org/experimental/website
We encourage community feedback! We are collecting feedback through the
following ticket: https://github.com/ooni/explorer/issues/525
### Other OONI Explorer improvements
We also made progress on the activities documented through the following
tickets:
* Adding more end-to-end testing of OONI Explorer:
https://github.com/ooni/explorer/issues/490 &
https://github.com/ooni/explorer/pull/493
* UI improvements: https://github.com/ooni/explorer/issues/511
* Upgrading to the latest ooni-components:
https://github.com/ooni/explorer/issues/509
* Adding nivo-toy to OONI Explorer:
https://github.com/ooni/explorer/pull/514
* Fixing issues in WhatsApp measurement pages:
https://github.com/ooni/explorer/issues/406
* Linking to measurements on OONI Explorer Country pages:
https://github.com/ooni/explorer/issues/338
## OONI backend
Throughout November 2020, we worked on the following backend activities:
* Created documentation, diagrams, dashboards, and runbooks
* Monitored 500 status code errors
* Looked into missing Telegram measurements
* Changed the rate limiting
* Wrote tooling to manage CI/CD deployments on test and prod hosts
* Implemented a backup/export of the fastpath table into S3
* Wrote a measurement_forwarder
* Deployed an event detector on test
* Monitored coverage before switching all traffic to test-list/urls from
ams-pg
* Fixed Nginx logging and ipaddr truncation
* Implemented API that allows OONI Probes to check-in before running tests
* Implemented and deployed the reactive test-list
* Added a dashboard to monitor the reactive test-list
* Added 2 new test types to the OONI API
* Tested the persistent journald
* Implemented a deployer to support OSX
## Internet Shutdown Measurement Training for Advocates
Throughout November 2020, we continued to facilitate (as a lead partner)
Internews’ Internet Shutdown Measurement Training for human rights
advocates in Sub-Saharan Africa.
During the week of 2nd November 2020, we helped facilitate the
synchronous training provided by M-Lab for the module on “Measuring
Internet Performance”.
During the week of 9th November 2020, we helped facilitate the
synchronous training for the module on “Contextualizing Your Data: Using
Local Insight and Qualitative Research”.
During the week of 16th November 2020, we helped facilitate the
synchronous training provided by Access Now for the module on “Using
Measurements in Advocacy”.
During the final week of November 2020, we helped training participants
prepare for their group presentations (at the end of the training
program on 1st December 2020).
Throughout the training program, we supported the training participants
by sharing relevant resources, addressing questions, reviewing their
homework assignments, and helping them prepare their projects.
### Created post-training survey
We created a (second, overall) survey for the participants of Internews’
Internet Shutdown Measurement Training program.
This survey was distributed towards the end of the 6-week training
program with the goal of collecting participant feedback on:
* The overall training program
* Each of the training modules (speaker and guest speaker sessions)
* The quality of the training mentorship
* Their intra-group collaboration
* The homework assignments
* How future training programs can be improved
Through this post-training survey, we also aimed to gauge participant
knowledge and skills acquired throughout the training program, in order
to evaluate its effectiveness.
## Created videos for micro-course on internet censorship
We created 3 short videos (as well as relevant slides and resources) for
an online micro-course on internet censorship.
Each of these videos cover the following topics:
* What is Internet Censorship?
* The Problem of Internet Censorship
* How to Detect Internet Censorship with OONI Probe
We expect to see these resources published by community members in 2021.
## Worked on research report on LGBTIQ website blocking
In collaboration with OutRight Action International and the Citizen Lab,
we have been working on a joint research report which examines the
blocking of LGBTIQ websites in 6 countries based on OONI data.
Throughout November 2020, we continued to edit the report.
## Collaboration with Netalitica on test lists
Netalitica researchers continued to update the Citizen Lab test lists,
and we reviewed (and shared feedback on) the updates to the Zimbabwean
and Algerian test lists.
We also made updates to several other test lists based on URLs shared by
community members:
https://github.com/citizenlab/test-lists/pull/682
https://github.com/citizenlab/test-lists/pull/694
https://github.com/citizenlab/test-lists/pull/684
## Data analysis for Azerbaijan Internet Watch
In support of our partner, Azerbaijan Internet Watch, we analyzed OONI
measurements collected from Azerbaijan between September 2020 to
November 2020.
In recent months, OONI measurements show that a number of social media
services started presenting signs of blocking in Azerbaijan (in addition
to the ongoing blocking of media websites) over the last months as well.
We documented the findings of our analysis in a report (including
relevant charts), which we shared with Azerbaijan Internet Watch.
## Preparing new Partnerships page for the OONI website
We have created a new page for the OONI website which features all of
our partners, highlights their work, links to their websites, and shares
some of the research reports and projects that we have collaborated on.
We plan to publish this page in December 2020.
In preparation, we reached out to our partners requesting their feedback
on this new page and we edited accordingly.
## Creating a new Donate page for the OONI website
We started working on creating a new Donate page for the OONI website.
Our work on this front involved the design and creation of relevant
mockups and frontend development.
## Community use of OONI data
### OTF Fellow research report on information controls in Myanmar
Over the last year, we had the opportunity to collaborate with and
support Phyu Phyu Kyaw, an OTF Information Controls Fellow, who
investigated surveillance and internet censorship (through the use of
OONI Probe and OONI data) in Myanmar.
More specifically, we supported Phyu Phyu Kyaw with OONI data analysis
(and relevant charts) on cases of internet censorship in Myanmar.
Phyu Phyu Kyaw’s report is published here:
https://www.opentech.fund/news/information-controls-unprotected-legal-landscape/
### Internews community needs assessment report
Internews carried out a study that aimed to document community needs
with respect to advocating against internet shutdowns. Their report,
based on this study, is available here:
https://static1.squarespace.com/static/5a9efdd2f2e6b149480187ea/t/5fbd37cda282d931e4d38c91/1606236110691/Optima+-+Needs+Assessment+Report+2020.pdf
Internews’ research found that the vast majority of advocates who
participated in this study reported that (out of several network
measurement tools) they are mainly familiar with (and have experience
using) OONI Probe. The community feedback collected through this study
is valuable as it will help inform the development and improvement of
our work.
## Community activities
### Internet Governance Forum (IGF)
We participated in the Internet Governance Forum (IGF) 2020 (which took
place entirely online) by hosting an OONI e-booth between 2nd to 13th
November 2020. We also presented OONI during the IGF Village virtual tours.
Through the OONI e-booth, we aimed to engage participants with OONI’s
censorship measurement tools and datasets, share resources, address
questions, and receive community feedback (through live consultations).
Information about the OONI e-booth at the IGF Village is available here:
https://www.intgovforum.org/multilingual/content/igf-2020-village-booth-60-open-observatory-of-network-interference-ooni
### State of the Onion 2020
On 16th November 2020, OONI’s Maria presented OONI during the Tor
Project’s annual State of the Onion 2020:
https://blog.torproject.org/state-of-the-onion-2020
More specifically, we shared OONI highlights from 2020, as well as
upcoming OONI projects for 2021. The State of the Onion 2020 event can
be viewed here: https://www.youtube.com/watch?v=IyWyTypRGWQ
### OONI Community Meeting
On 24th November 2020, we hosted the monthly OONI Community Meeting on
our Slack channel (https://slack.ooni.org/), during which we discussed
the following topics:
1. Brief updates from the OONI-verse
2. Open source and reliable auditing frameworks
3. Integrating the new DNSCheck experiment into the OONI Probe mobile
and desktop apps
4. Requirements for running and shipping new experimental tests: Request
for community feedback
## Userbase
In November 2020, 3,368,016 OONI Probe measurements were collected from
4,679 networks in 195 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
We examined this drop in measurement coverage (in comparison to previous
months), and we found that it is explained primarily by the following 2
reasons:
* We recently stopped processing and publishing AS0 measurements,
particularly because measurements without an ASN have very limited value;
* Several legacy probes (which contributed large volumes of stable
measurements) were discontinued over the last few months.
That said, we aim to boost global measurement coverage over the next
months once regular automatic testing is shipped as part of the OONI
Probe mobile app. We also plan to release official Linux and macOS
packages for OONI Probe, which will serve as a replacement for legacy
probe users.
~ OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
More information about the tor-project
mailing list