[tor-project] OONI Monthly Report: March 2020
Maria Xynou
maria at openobservatory.org
Thu Apr 2 13:40:12 UTC 2020
Hello,
Throughout March 2020, the OONI team worked on the following 2
(two-week) sprints:
* Flounder Sprint (2nd March 2020 - 15th March 2020)
* Ariel Sprint (16th March 2020 - 29th March 2020)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Details and highlights are available in this report below.
## OONI Probe mobile app
We released OONI Probe mobile 2.3.0!
* Android: https://github.com/ooni/probe-android/releases/tag/v2.3.0
* iOS: https://github.com/ooni/probe-ios/releases/tag/v2.3.0
Highlights include:
1. You can re-test URLs. This new feature can be useful if you want to
collect more evidence on potential blocking, or if a test fails.
2. You can test many more websites in one go! By disabling the “Limit
test duration” option (of the website settings), you can test all of the
URLs included in the Citizen Lab's global and country-specific test lists.
This release also fixed a bug that led to WhatsApp false positive test
results (once WhatsApp made changes to its infrastructure), and includes
a series of other improvements documented here:
https://github.com/ooni/probe-android/releases/v2.3.0
To address certain infrastructure issues that emerged following the
release of OONI Probe mobile 2.3.0, we released OONI Probe mobile 2.3.1
with a fix: https://github.com/ooni/probe-android/releases/tag/v2.3.1
We also released OONI Probe mobile 2.3.2 with further UX improvements
and bug fixes: https://github.com/ooni/probe-android/releases/tag/v2.3.2
In March 2020, we also started working on adding support for automated
testing: https://github.com/ooni/probe-android/pull/304
Finally, we planned the backlog and roadmap of OONI Probe mobile to
better determine the next steps of OONI Probe mobile development:
https://github.com/ooni/probe/issues/977
## Research on push notifications
In order to add support to the OONI Probe mobile app to send push
notifications, we have been researching the use of Countly as a
potential frontend and backend solution for push notifications.
As part of this, we:
* Enabled analytics on countly: https://github.com/ooni/probe/issues/1002
* Tested countly to see whether we can use push notifications for
sending OONI Run links: https://github.com/ooni/probe/issues/1055
* Checked how to set-up and use the countly rich notification plugin:
https://github.com/ooni/probe/issues/1056
* Explored whether the countly community edition is sufficient to meet
our needs: https://github.com/ooni/probe/issues/1057
We also researched how to design the push notification backend, as
documented through the following ticket:
https://github.com/ooni/backend/issues/339
## OONI Probe desktop app
Throughout March 2020, we continued to make improvements to the OONI
Probe desktop app in preparation for the upcoming public launch.
More specifically, we:
* Made a new release candidate for the OONI Probe desktop app (including
newly translated strings for the circumvention tool tests):
https://github.com/ooni/probe-desktop/releases/tag/v3.0.0-rc.6
* Made UX improvements to the Dashboard and measurement pages:
https://github.com/ooni/probe/issues/1047 &
https://github.com/ooni/probe/issues/1042
* Integrated the animation for the circumvention tool tests:
https://github.com/ooni/probe/issues/1063
* Investigated the failures in the auto-update system and added better
exception logging: https://github.com/ooni/probe/issues/1008
* Aligned the middlebox test results in the Test Results page:
https://github.com/ooni/probe/issues/1049
* Fixed the markdown in the Psiphon measurement pages:
https://github.com/ooni/probe/issues/1051
* Edited the copy in the Websites card:
https://github.com/ooni/probe/issues/1048
* Fixed OONI Explorer links: https://github.com/ooni/probe/issues/1041
* Fixed the z-index and draggability of Hero on measurement pages:
https://github.com/ooni/probe/issues/1033
* Fixed the scrolling on measurement pages:
https://github.com/ooni/probe/issues/959
* Fixed the sorting of columns in the Tor test details page:
https://github.com/ooni/probe/issues/1007
## Creating official Linux packages for OONI Probe
To better support the Linux OONI community (which contributes the
largest volume of measurements from stable vantage points), we aim to
create official Linux packages for OONI Probe.
To this end, we started doing some initial Debian packaging for Linux in
March 2020 and we created a prototype (see:
https://github.com/ooni/probe-cli/pull/117), which is available here:
https://github.com/ooni/probe/issues/1061
## OONI Probe measurement engine
OONI Probe has relied on the MaxMind database (https://www.maxmind.com/)
for IP geolocation, but they recently changed their license.
In March 2020, we continued to work on replacing the MaxMind ASN
database following the license change:
https://github.com/ooni/probe-engine/issues/336
## Making the OONI Probe apps rely entirely on the OONI Probe golang engine
We continued to make progress on making the OONI Probe mobile and
desktop apps rely entirely on the golang based probe-engine.
To this end, we completed the work in the following tickets:
https://github.com/ooni/probe-engine/issues/348
https://github.com/ooni/probe-engine/issues/350
https://github.com/ooni/probe-engine/issues/351
https://github.com/ooni/probe-engine/issues/353
https://github.com/ooni/probe-engine/issues/362
https://github.com/ooni/probe-engine/issues/399
https://github.com/ooni/backend/issues/369
https://github.com/ooni/probe/issues/1030
## Improving censorship circumvention tool methodologies
We are improving our censorship measurement methodologies by including
metrics that are related to measuring the performance of circumvention
tools.
We aim to add support for running the NDT speed and performance test
over circumvention tools (Tor, Psiphon, obfs4proxy). This will allow us
to determine at what throughput (for example, a 1Mbit download) the tool
is able to perform on the network in question, and to better
characterize the circumvention tool’s performance.
In March 2020, we started off by replacing NDT with the next generation
ndt7 speed test in the OONI Probe Command Line Interface (CLI). This
work is available through the following pull request:
https://github.com/ooni/probe-cli/pull/119
## Expanding OONI Probe measurement methodologies
As part of our ongoing efforts to expand OONI Probe measurement
methodologies, we made improvements to the implementation of our SNI
experiment, which involved addressing feedback shared by an engineer at
Jigsaw (Vinicius Fortuna): https://github.com/ooni/probe-engine/issues/309
## Prototype of system for probe URL selection
Throughout March 2020 -- during the OONI Team Meeting and during our
weekly backend and frontend development calls -- we did a lot of
brainstorming in order to determine what an MVP for the URL selection
prototype should look like.
Our plan for the MVP is documented in the following github ticket:
https://github.com/ooni/backend/issues/361
We also documented several issues related to the creation of the prototype:
https://github.com/ooni/backend/issues/320
https://github.com/ooni/backend/issues/361
https://github.com/ooni/backend/issues/321
https://github.com/ooni/ooni.org/issues/431
https://github.com/ooni/ooni.org/issues/430
https://github.com/ooni/backend/issues/322
https://github.com/ooni/backend/issues/221
https://github.com/ooni/backend/issues/207
https://github.com/ooni/probe/issues/930
https://github.com/ooni/explorer/issues/392
https://github.com/ooni/explorer/issues/391
## Data analysis and pipeline work necessary for extracting per site metrics
In March 2020, we started experimenting with ClickHouse to evaluate
whether it could be suitable for the more advanced batch analysis
required for extracting website metrics. So far, the results are quite
promising. The cursory findings of this research are documented here:
https://github.com/ooni/backend/issues/380
We also did a lot of database tuning and troubleshooting related to
adding counter tables which support aggregating per website metrics:
https://github.com/ooni/backend/issues/371
Furthermore, we:
* Wrote an analysis script to close active database connections and fix
a connections issue: https://github.com/ooni/pipeline/pull/306
* Made progress on keeping the fast-path rows in the measurement table:
https://github.com/ooni/backend/issues/233
## OONI Probe orchestration logic for circumvention tool testing
As part of our work on developing OONI Probe orchestration logic that is
specific to circumvention tool testing, we:
* Added backend support for provisioning circumvention tool test
configurations to probes: https://github.com/ooni/orchestra/issues/68
* Made it possible to instruct probes in near real-time to test a
specific circumvention tool with a particular configuration:
https://github.com/ooni/probe/issues/889
## OONI Explorer
Throughout March 2020, we made a few improvements to OONI Explorer.
More specifically, we:
* Fixed the anomaly indicator to ensure that the search results are
consistent with the anomaly status in the measurement pages:
https://github.com/ooni/explorer/issues/360
* Made measurement pages resilient to the absence of nullable data:
https://github.com/ooni/explorer/issues/414
* Handled cases where tcp_connect test keys are missing:
https://github.com/ooni/explorer/pull/412
* Wrote copy for the new Psiphon test results:
https://github.com/ooni/explorer/issues/419
* Wrote copy for the new Tor test results:
https://github.com/ooni/explorer/issues/418
* Opened tickets based on community feedback:
https://github.com/ooni/explorer/issues?q=is%3Aopen+is%3Aissue+label%3A%22user+feedback%22
## Quality assurance and testing of OONI software
As part of our ongoing efforts to increase the quality of OONI software
components, we carried out the following activities in March 2020:
* Created an end-to-end testing strategy for OONI Explorer:
https://github.com/ooni/explorer/issues/159
* Added end-to-end tests for the OONI Probe desktop app:
https://github.com/ooni/probe/issues/1062
* Tested the OONI Probe mobile app 2.3 release:
https://github.com/ooni/ooni.org/issues/398
We also made the improvements to the probe-engine, as documented through
the following tickets:
https://github.com/ooni/probe-engine/issues/310
https://github.com/ooni/probe-engine/issues/315
https://github.com/ooni/probe-engine/issues/317
https://github.com/ooni/probe-engine/issues/327
## OONI infrastructure
As part of our efforts to improve our server infrastructure and the
performance of reliant tools, we:
* Dealt with an incident affecting API slowdown and frequent 500 errors:
https://github.com/ooni/backend/issues/371
* Dropped some unused hosts once they stopped receiving traffic:
https://github.com/ooni/backend/issues/309
* Investigated slow OONI Explorer / OONI API queries:
https://github.com/ooni/backend/issues/327
* Investigated why certain OONI Explorer measurements were unreachable:
https://github.com/ooni/backend/issues/329
* Added supported for automatically building docker images as part of
the CI process: https://github.com/ooni/backend/issues/346
* Performed VACCUM of the metadb:
https://github.com/ooni/backend/issues/354
* Stopped doing full table scans in some places to improve API
performance: https://github.com/ooni/backend/issues/356
## OONI website
In preparation for the upcoming public launch of the OONI Probe desktop
app, we worked on re-designing the Install page of our website. A
community member generously volunteered to write the code for
implementing the new Install page based on our design mockups. This work
is available here: https://github.com/ooni/ooni.org/pull/446
We also wrote and published the following:
* Tor test description: https://ooni.org/nettest/tor/
* Psiphon test description: https://ooni.org/nettest/psiphon/
Furthermore, we made improvements to how images are handled in the OONI
website: https://github.com/ooni/ooni.org/issues/390
## Published report on the temporary blocking of the Farsi edition of
Wikipedia in Iran
We published a report on the temporary blocking of the Farsi language
edition of Wikipedia in Iran:
https://ooni.org/post/2020-iran-blocks-farsi-wikipedia/
OONI measurements showed that the Farsi edition of Wikipedia
(fa.wikipedia.org) was blocked on at least 6 networks in Iran between
2nd to 3rd March 2020.
We also found the Farsi edition of Wikinews (fa.wikinews.org) blocked in
Iran, though we're being told that it's been blocked for several years now.
Both Farsi Wikipedia and Wikinews were blocked by means of DNS tampering
and SNI filtering. In our report, we share relevant data.
## Published blog post encouraging the testing of COVID-19 websites
Ensuring a free and open internet during emergencies is important, but
access to COVID-19 information has already been censored in China and
Venezuela.
We therefore published a blog post encouraging community participation
in a global measurement campaign to increase testing coverage of
COVID-19 relevant websites.
This blog post (providing relevant instructions for testing) is
available here: https://ooni.org/post/covid-19-ooni-measurement/
We also embedded a “COVID-19” button on the OONI website to enable OONI
Probe testing of COVID-19 websites: https://ooni.org/get-involved/run/
## Year in Review: OONI in 2019
We published a blog post sharing highlights from the OONI-verse over the
last year, as well as some of the great ways that community members made
use of OONI Probe and OONI data as part of their work.
The “Year in Review: OONI in 2019” post is available here:
https://ooni.org/post/ooni-in-2019/
OONI highlights from 2019 were also discussed in a separate blog post
written and published by VPNCompare:
https://www.vpncompare.co.uk/ooni-censorship-2020s/
## Published blog post on OONI highlights between 2017-2019
We wrote a blog post that shares OONI highlights from 2017 to 2019 that
were made possible thanks to OTF support.
This blog post was published on the OTF website:
https://www.opentech.fund/news/ooni-highlights-2017-2019/
We also cross-posted this blog post on the OONI website:
https://ooni.org/post/ooni-highlights-2017-2019/
## OONI Team Meeting
We held a 4-day OONI Team Meeting between 10th to 13th March 2020.
We were originally planning on facilitating this meeting in person in
Rome. However, due to the escalating impact of COVID-19 (coronavirus)
and the national lockdown of Italy (which started on 10th March 2020),
we cancelled travel arrangements (for OONI team members not based in
Rome) and facilitated the entire 4-day meeting remotely, online.
During our team meeting, we refined our roadmap for the next 6 months
and held various meetings/sessions that enabled us to discuss plans and
next steps for the various projects in the OONI-verse.
Below we share the sessions that we held during our 4-day team meeting.
### Day 1
* Discussion of team meeting agenda
* Time for reflection: What worked well and what didn’t work well over
the last 6 months
* Test vectors and technical documentation
* Expanding OONI Probe methodologies
* Data analysis needs: Building capacity
* Data analysis required for extracting website metrics
### Day 2
* What do you want OONI to be?
* Adding support for OONI Probe push notifications
* Brainstorming on URL prioritization
* Server infrastructure: Next steps
* Presenting website metrics on OONI Explorer
* OONI Explorer: Next steps
### Day 3
* Improving OONI Run
* OONI Probe Mobile: Next steps
* OONI Probe Desktop: Next steps
* Identifiers of measurements (OOID) and in defense of the report ID
* OONI data processing pipeline design
### Day 4
* OONI Team Roadmap for 2020
* Fundraising priorities
Most of Day 4 of the team meeting was spent creating a detailed roadmap
for all of our projects until December 2020.
We expect to revise our roadmap during our next team meeting which we
plan to have in about 6 months from now (September or October 2020).
## Fundraising
In March 2020, we were invited by the Ford Foundation to submit a
proposal to seek support for hosting the next OONI Partner Gathering and
other community-related work.
We previously submitted a proposal to the OTF Community Lab to seek
support for hosting the OONI Partner Gathering, and in March 2020 we
addressed their latest feedback.
## Test list updates
Throughout March 2020, we:
* Updated the Venezuelan test list to include reportedly blocked
coronavirus-related sites: https://github.com/citizenlab/test-lists/pull/591
* Updated the Global test list to include many coronavirus-related
websites (particularly data maps):
https://github.com/citizenlab/test-lists/pull/594
* Updated the Global & Saudi Arabian test lists to include LGBTQI sites
shared by OutRight International:
https://github.com/citizenlab/test-lists/pull/593
* Updated the Myanmar test list to include URLs from a new blocklist
(shared by an OTF Information Controls Fellow):
https://github.com/citizenlab/test-lists/pull/596 &
https://github.com/citizenlab/test-lists/pull/597
## Data analysis
Throughout March 2020, we performed OONI data analysis for the following:
* Azerbaijan Internet Watch: Analysis of measurements collected from
Azerbaijan
* Small Media: Analysis of OONI measurements collected from Lebanon &
Rwanda, in support of relevant stakeholder submissions to the Universal
Periodic Review (UPR): https://github.com/ooni/ooni.org/issues/435
Based on these analyses, we wrote reports explaining and interpreting
the findings, which we shared with Azerbaijan Internet Watch and Small
Media.
## Supported OTF Fellow
Over the last year, we have hosted an OTF Information Controls Fellow
who has been researching internet censorship in certain states of India.
In preparation for the publication of their research report, we
continued to review it, share suggested copy edits and other feedback.
We also made improvements to the data visualizations included in their
research report: https://github.com/ooni/ooni.org/issues/441
## Google Summer of Code (GSoC)
OONI is participating in Google Summer of Code (GSoC) under The Tor
Project and the DIAL Open Source Center (OSC). In March 2020, prospect
GSoC students interested in working on OONI projects over the summer
reached out to us.
Below are some of the projects that we suggested to the prospect GSoC
students:
* OONI Probe Desktop support for OONI Run and custom URLs,
https://community.torproject.org/gsoc/ooni-probe-desktop-custom-urls/
* OONI Probe network experiments,
https://community.torproject.org/gsoc/ooni-probe-experiments/
* OONI Explorer: Improvements to testing,
https://community.torproject.org/gsoc/ooni-explorer-testing/
* OONI Explorer findings: Improvements related to social media sharing,
https://community.torproject.org/gsoc/ooni-explorer-findings/
## IEEE article discussing OONI
OONI’s Arturo was interviewed by IEEE to explain and discuss OONI.
IEEE subsequently published an article that presents OONI and discusses
how our project measures internet censorship around the world.
This article is available here:
https://spectrum.ieee.org/tech-talk/telecom/internet/how-to-detect-a-governments-hand-behind-internet-shutdowns
## Community activities
### RIPE hackathon
We contributed project ideas to the online RIPE hackathon:
https://labs.ripe.net/Members/becha/hackathons-in-the-time-of-corona
We also participated in the hackathon video-chat (23rd March 2020) and
offered the #ooni-dev channel (on Slack and IRC) as a space for online
communication and coordination during the hackathon.
### OONI Community Meeting
On 31st March 2020, we facilitated the monthly OONI Community Meeting on
our Slack channel (https://slack.ooni.org/), during which we discussed
the following topics:
1. Updates on blocked Biafra websites in Nigeria
2. Testing a custom list of URLs with the OONI Probe CLI
3. The future of automatic OONI Probe testing on desktop and Raspberry Pis
4. Storage data format that is easier to consume
5. Faster event analysis
## Userbase
In March 2020, 8,214,072 OONI Probe measurements were collected from
5,719 networks in 212 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
http://explorer.ooni.org/
~ The OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200402/06191fb5/attachment.sig>
More information about the tor-project
mailing list