[tor-project] OONI Monthly Report: June 2020
Maria Xynou
maria at openobservatory.org
Wed Jul 22 10:06:30 UTC 2020
Hello,
Throughout June 2020, the OONI team worked on the following sprints:
* Sprint 14 - Ponyo (1st June 2020 - 7th June 2020)
* Sprint 15 - Globster (8th June 2020 - 21st June 2020)
* Sprint 16 - Neon (22nd June 2020 - 30th June 2020)
Our work can be tracked through the various OONI GitHub repositories:
https://github.com/ooni
Highlights are shared in this report below.
## Released OONI Probe Mobile 2.5
We released OONI Probe Mobile 2.5!
Android: https://github.com/ooni/probe-android/releases/tag/v2.5.0
iOS: https://github.com/ooni/probe-ios/releases/tag/v2.5.0
Highlights from the latest release include:
1) Circumvention tool testing: The latest release includes OONI Probe
tests for measuring the blocking of Tor & Psiphon!
2) 5 new languages: Thanks to the Localization Lab community, the OONI
Probe mobile app is now also available in Indonesian, Thai, Romanian,
Slovak, and Icelandic (in total, the OONI Probe mobile app is now
available in 20 languages!)
As part of the iOS release, we also rewrote NDT to increase its
reliability and measurement accuracy, and wrote the Telegram, NDT, and
DASH tests in golang.
## OONI Probe desktop app
In June 2020, we made two OONI Probe desktop app releases:
* OONI Probe Desktop 3.0.2:
https://github.com/ooni/probe-desktop/releases/tag/v3.0.2
* OONI Probe Desktop 3.0.3:
https://github.com/ooni/probe-desktop/releases/tag/v3.0.3
These releases feature bug fixes and other improvements based on
community feedback.
## Making the OONI Probe apps rely entirely on the golang engine
As part of our ongoing efforts to make the OONI Probe apps rely entirely
on the golang engine, we:
* Used JCenter to fetch probe-engine on Android:
https://github.com/ooni/probe/issues/1191
* Updated to the latest version of probe-cli/probe-engine:
https://github.com/ooni/probe/issues/1178
* Made a series of updates: https://github.com/ooni/probe-engine/issues/689
## Adding support for configuring push notifications
Our goal is to be able to send push notifications to OONI Probe mobile
app users through the use of Countly, which enables us to include links
(such as OONI Run links for coordinated OONI Probe testing) in push
notifications.
To this end, we evaluated how to manage the collection of app usage
metrics, crash reports, and privacy settings, as documented through the
following ticket: https://github.com/ooni/probe/issues/1182
We also made progress on implementing countly on Android:
https://github.com/ooni/probe-android/pull/341
## Adding support in OONI Probe for availability testing of the
circumvention tools Tor, obfs4proxy, and Psiphon
We have completed our work related to adding support in OONI Probe for
availability testing of Tor, obfs4proxy, and Psiphon. The new Tor and
Psiphon tests are available through the recently launched OONI Probe
desktop app (https://ooni.org/install/desktop).
In June 2020, we also implemented the STUN server reachability
experiment: https://github.com/ooni/probe-engine/issues/243
## Improving censorship circumvention tool methodologies by including
performance metrics
We completed our (short-term) work on improving our methodologies for
measuring censorship circumvention tools by measuring the performance of
those tools as well.
As part of our final relevant activities, we:
* Recorded website accessibility performance metrics when using Psiphon:
https://github.com/ooni/probe-engine/issues/404
* Improved the data format versioning:
https://github.com/ooni/probe-engine/issues/423
* Allowed using an external tor binary to run ndt7, DASH, and urlgetter
over it: https://github.com/ooni/probe-engine/issues/587
## Developing OONI Probe orchestration logic that is specific to
circumvention tool testing
We completed our work related to developing OONI Probe orchestration
logic that is specific to circumvention tool testing.
As part of our final relevant activities, we:
* Wrote an integration test for private Tor bridges:
https://github.com/ooni/probe-engine/issues/721
* Stripped private bridge addresses:
https://github.com/ooni/probe-engine/issues/643
* Included the country code in the Tor query string:
https://github.com/ooni/probe-engine/issues/629
## Making OONI Probe’s reporting logic more resilient to censorship
We completed our (short-term) efforts related to making OONI Probe’s
reporting logic more resilient to censorship.
As part of our final relevant activities, we:
* Discussed requirements for detecting the blocking of OONI services in
the pipeline: https://github.com/ooni/backend/issues/417
* Created new domain names: https://github.com/ooni/backend/issues/424
* Implemented a prototype of Probe Services blocking detection:
https://github.com/ooni/backend/issues/428
* Implemented failover between available OONI data centers:
https://github.com/ooni/probe-engine/issues/621
* Used best available OONI data center for orchestration and other
services: https://github.com/ooni/probe-engine/issues/651
We did an analysis (https://github.com/ooni/probe/issues/886), using the
fact that we added support for probes to failover to cloudfront
collectors, and we did not notice any probes failing over using this method.
We have not identified any cases of blocking of OONI infrastructure.
That said, we will continue to do more investigations and eventually
automate the process.
## Improving URL testing
### Implementing beta quality backend logic for prioritising URLs
As part of our work on implementing beta quality backend logic for the
prioritization of URLs, we worked on refactoring and improving our URL
prioritization backend (and on creating an internal Grafana dashboard).
See: https://github.com/ooni/pipeline/pull/321
Moreover, we started adding an haproxy server in front of the URL
prioritisation backend so that we are able to start using it in
production for a percentage of the probes out there. See:
https://github.com/ooni/sysadmin/pull/446.
### Implementing beta quality frontend to support management of the URL
priorities
In order to implement a beta quality frontend to support the management
of URL priorities, we started off by reviewing the relevant priorities.
As part of this process, we documented the next steps in the following
tickets:
https://github.com/ooni/backend/issues/429
https://github.com/ooni/ooni.org/issues/524
## Analyzing data to extract website metrics
As part of our ongoing data analysis work to extract website metrics, we
published an aggregation API endpoint and started working on creating a
page to try out the new aggregation API. The goal of this page is to
query the API for different kinds of data and to render charts based on
that data. Eventually, these charts will be integrated in the country
pages of OONI Explorer. See: https://github.com/ooni/explorer/issues/463
Our initial experiment for trying out the new aggregation API is
available here:
https://codesandbox.io/s/nivobar-extra-layers-pbgg4?file=/src/index.js
## Presenting website measurements to account users
We are working towards enabling community members to have access to
advanced and database-heavy views of website-centric measurements
(through accounts that we will create). To this end, we started off by
discussing and deciding what it means to have an account within the
context of the various OONI platforms:
https://github.com/ooni/ooni.org/issues/434
As part of this research, we wrote an internal document outlining our
plans for implementing accounts and authentication to OONI services. We
also documented the use cases (user stories) for these accounts,
feedback and community requests pertaining to accounts (as communicated
to us through a pull in a previous community meeting, as well as through
other long-term interactions with community members), as well as
implementation details and requirements.
We also worked on creating mock-ups for website-centric views:
https://github.com/ooni/explorer/issues/472
## Improving our server infrastructure
As part of our ongoing work to improve our server infrastructure, we
improved the performance of the counters table
(https://github.com/ooni/backend/issues/411) and we made improvements to
the monitoring of our infrastructure
(https://github.com/ooni/backend/issues/427).
## Testing and quality assurance
We made progress on improving the quality of our software through better
testing and quality assurance procedures. We performed data analysis
based on specific measurements
(https://github.com/ooni/probe-engine/issues/662) and implemented
changes (described below in detail) to simplify subsequent data analysis
attempts. We also further evaluated the code for selecting the closest
OONI probe service API among the available data centers (HK, MIA, AMS),
and extended this functionality to OONI orchestration.
Specifically, we:
* Performed experiments and data analysis based on measurements
performed in India (which resulted in a blog post):
https://github.com/ooni/probe-engine/issues/662
* Simplified OONI data analysis by ensuring that the resolver is added
to the measurement: https://github.com/ooni/probe-engine/issues/642
* Simplified OONI data analysis by setting the DNS cache to null when
empty: https://github.com/ooni/probe-engine/issues/658
* Improved urlgetter measurements:
https://github.com/ooni/probe-engine/issues/656
* Used best data center providing the probe service API for
orchestration and other services:
https://github.com/ooni/probe-engine/issues/651
* Wrote an integration test for private Tor bridges:
https://github.com/ooni/probe-engine/issues/721
Throughout June 2020, we focused quite a bit on improving the data
quality of OONI measurements.
## Published report on the blocking of DNS over TLS in Iran
In June 2020, we published a research report, titled "DNS over TLS
blocked in Iran", which is available here:
https://ooni.org/post/2020-iran-dot/
We investigated whether DoT works in Iran by gathering a list of 31
well-known DoT endpoints and running experiments from four distinct
Iranian mobile and fixed-line Internet Service Providers (ISPs): MCI,
TCI, Irancell, and Shatel.
We discovered that:
* 57% of the endpoints are blocked on a least one ISP;
* the blocking is not implemented uniformly across ISPs;
* most blocking happens by interfering with the TLS handshake;
* in some cases TLS handshake blocking seems to depend on the SNI, while
in other cases it seems to depend strictly on the TCP endpoint being used;
* forcing TLSv1.3 does not change the rate of successful TLS handshakes
compared to letting the server choose a TLS version between v1.0 and v1.3.
In our report, we share details from our experiments and findings. This
research is part of our ongoing efforts to expand and improve upon our
measurement methodologies.
## Published report on TLS blocking in India
In collaboration with India’s Centre for Internet and Society (CIS), we
co-published a research report investigating TLS blocking in India.
This report is available here:
https://ooni.org/post/2020-tls-blocking-india/
This investigation sought to understand whether there were cases of TLS
blocking that were not only caused by the value of the Server Name
Indication (SNI) field in the ClientHello TLS message, but also by the
destination IP address. This was part of our efforts to expand our SNI
blocking methodology (discussed here:
https://ooni.org/post/2020-iran-sni-blocking/).
To this end, we wrote and ran a series of experiments (that will
eventually be integrated into the OONI Probe measurement engine) to
measure the blocking of four domains (facebook.com, google.com,
collegehumor.com, and pornhub.com) on three popular Indian ISPs: ACT
Fibernet (fixed line), Bharti Airtel, and Reliance Jio (mobile).
We recorded SNI-based blocking on both Bharti Airtel and Reliance Jio.
We also discovered that Reliance Jio blocks TLS traffic not just based
on the SNI value, but also on the web server involved with the TLS
handshake.
We also noticed that ACT Fibernet’s DNS resolver directs users towards
servers owned by ACT Fibernet itself. Such servers caused the TLS
handshake to fail, but the root cause of censorship was the DNS.
We also found that one of the tested endpoints (for
collegehumor.com:443) does not allow establishing a TCP connection from
several vantage points and control measurements. Yet, in Reliance Jio,
we saw cases where the connections to such endpoints completed
successfully and a timeout occurred during the TLS handshake. This is
likely caused by some kind of proxy that terminates the TCP connection
and performs the TLS handshake.
## Expanding OONI Probe measurement methodologies
Our aforementioned research reports on DNS over TLS blocking in Iran
(https://ooni.org/post/2020-iran-dot/) and TLS blocking in India
(https://ooni.org/post/2020-tls-blocking-india/) document our ongoing
efforts to expand our measurement methodologies.
The experiments that we design and run as part of these research efforts
will eventually be integrated into OONI Probe and shipped as part of our
apps.
Some of our efforts on expanding our measurement methodologies are also
documented through the following tickets:
https://github.com/ooni/probe-engine/issues/624
https://github.com/ooni/probe-engine/issues/576
https://github.com/ooni/probe-engine/issues/659
## Published report of OTF Information Controls Fellow
Over the last year, OONI has had the opportunity to serve as the host
organization of OTF Information Controls Fellow, Chinmayi S K
(https://www.opentech.fund/about/people/chinmayi-sk/).
In June 2020, we published Chinmayi's research report, titled "Those
Unspoken Thoughts: A study of censorship and media freedom in Manipur,
India".
Summary of report:
https://ooni.org/post/2020-those-unspoken-thoughts-otf-fellow-report/
Full length report:
https://ooni.org/documents/those-unspoken-thoughts-otf-fellow.pdf
As part of her fellowship, Chinmayi researched internet censorship in
Manipur, India, and its effect on the womxn in the region. To
investigate internet censorship, Chinmayi ran OONI Probe in various
regions of India and analyzed relevant OONI measurements. To explore the
impact of internet censorship on womxn in the region and to investigate
restrictions to media freedom, she circulated a survey and carried out
interviews.
Throughout her fellowship, OONI provided research, editing, and data
analysis support. When we published Chinmayi’s report in June 2020, we
also assisted with relevant outreach activities.
## Published statement in support of the OTF
Following the firing of OTF’s leadership, we published a statement in
support of the OTF, explaining why the OTF is essential for internet
freedom.
Our statement is available here:
https://ooni.org/post/2020-06-19-save-internet-freedom-support-the-open-technology-fund/
## Collaboration with Azerbaijan Internet Watch
### Published guest blog post to encourage more OONI Probe testing in
Azerbaijan
To encourage more OONI Probe testing in Azerbaijan, we published a guest
blog post (in Azerbaijani) written by Arzu Geybullayeva of Azerbaijan
Internet Watch (with whom we partner).
This blog post is available here:
https://ooni.org/post/2020-azerbaijan-ooni-probe-testing/
The post provides step-by-step instructions for OONI Probe testing,
particularly with regards to the testing of media websites and
circumvention tools, which present signs of blocking based on OONI data
analysis.
### Data analysis
As part of our partnership with Azerbaijan Internet Watch, we analyzed
all OONI measurements collected from Azerbaijan between 1st January 2020
to 30th June 2020 (https://github.com/ooni/ooni.org/issues/543), based
on which we produced relevant charts and wrote a report.
## Collaboration with Netalitica
Netalitica researchers continued to do great work in reviewing and
updating the Citizen Lab test lists
(https://github.com/citizenlab/test-lists).
In June 2020, we reviewed URLs shared by Netalitica researchers and
added them to the Turkish test list:
https://github.com/citizenlab/test-lists/pull/642
We also made updates to the Azerbaijan
(https://github.com/citizenlab/test-lists/pull/640) and Hong Kong
(https://github.com/citizenlab/test-lists/pull/641) test lists, based on
URLs shared by community members.
## Press coverage
RESET interviewed the OONI team and published an article about OONI.
This article was published in:
* German:
https://reset.org/blog/ooni-software-spuert-menschenrechtsverletzungen-im-internet-06042020
* English:
https://en.reset.org/blog/ooni-free-software-designed-detect-human-rights-violations-internet-06172020
DARK Reading also published an article that mentions OONI Probe,
following an interview with OONI’s Arturo. This article is published
here:
https://www.darkreading.com/cloud/no-internet-access-amid-protests-heres-how-to-tell-whether-the-government-is-behind-it/d/d-id/1338169
## Community use of OONI data
### Psiphon Data Engine (PDE)
Over the last year, we have collaborated with Psiphon on integrating
OONI measurements into a new platform, called Psiphon Data Engine (PDE).
This platform displays OONI and M-Lab measurements, along with Psiphon
metrics, through an interactive dashboard.
In June 2020, Psiphon launched the PDE, which is available here:
https://psix.ca/
### Report on the blocking of Blogger in Lebanon
Through the use of OONI Probe and OONI data, SMEX examined the blocking
of Blogger in Lebanon.
They published a report based on their findings, which is available
here:
https://smex.org/ar/%d8%ad%d8%ac%d8%a8-%d8%a8%d9%84%d9%88%d8%ba%d8%b1-%d9%81%d9%8a-%d9%84%d8%a8%d9%86%d8%a7%d9%86-%d9%83%d9%8a%d9%81-%d8%aa%d8%b3%d8%aa%d9%85%d8%b1%d9%91-%d9%88%d8%b2%d8%a7%d8%b1%d8%a9-%d8%a7%d9%84%d8%a7/
### Report on potentially blocked media in Belarus
In June 2020, Human Constanta published a report examining several cases
of network interference in Belarus. As part of their study, they also
used OONI Probe to examine the blocking of several media websites.
Their report (which includes OONI findings) is available here:
https://humanconstanta.by/chto-proisxodilo-s-internetom-v-belarusi-19-iyunya/?fbclid=IwAR0evE1Gbt1-R7YYIXA_mcfTvaVk-C5ofuEFC_4jdrVC9kvl9OOHGH4gH1g
### 2019 Annual Report on Digital Rights in Venezuela
Our Venezuelan partners, IPYS Venezuela, published their 2019 annual
report on digital rights in Venezuela, which is available here:
https://ipysvenezuela.org/2020/05/17/desconexion-y-censura-reporte-anual-derechos-digitales-ipysve-2019/
This report discusses censorship findings that they discovered through
the use of OONI Probe and OONI data.
### Blocking of YouTube in Venezuela
Through the use of OONI Probe and OONI data, our local partners in
Venezuela reported the blocking of YouTube:
https://twitter.com/ipysvenezuela/status/1271196383463247872
## Community activities
### Internet Measurement Village 2020
Throughout June 2020, we organized and hosted the first Internet
Measurement Village (IMV) 2020. Information about the event is available
through our relevant blog post:
https://ooni.org/post/2020-internet-measurement-village/
The IMV is an online community event -- dedicated to discussions around
internet measurement -- that took place throughout June 2020, starting
on 10th June 2020 and ending on 3rd July 2020.
During this 4-week period, we hosted one session almost every day (on
weekdays), all of which were live-streamed on the OONI YouTube channel:
https://www.youtube.com/c/OONIorg
In total, we hosted 18 live-streamed sessions, covering a wide range of
internet measurement projects, as well as local censorship measurement
projects, advocacy efforts against internet shutdowns, and censorship
circumvention tool projects.
To encourage participation, we hosted a live chat on several platforms
(YouTube chat, Slack, IRC) and, on average, each presenter received
around 10 questions from the viewers. All questions discussed during
each session are available through this pad:
https://pad.riseup.net/p/imv2020-keep
Apart from hosting 18 live-streamed sessions, we also worked on
promoting each session through the creation of relevant social media
assets, as well as through outreach on multiple mailing lists and on all
our social media platforms (Twitter, Mastodon, Facebook, Instagram).
The Internet Measurement Village 2020 also featured an online social
event, the IMV2020 Distance Disco, hosted by Sandy Ordonez on the
following platform: https://distancedisco.nl/
At the end of the Internet Measurement Village 2020, we shared a survey
to collect community feedback and learn how we can host better events in
the future. Our survey is available here:
https://ooni.typeform.com/to/cPVT4ILB
As all the live-streamed sessions of the IMV will continue to live on
the OONI YouTube channel, we hope that these recordings will serve as a
valuable resource on internet measurement for the internet freedom
community.
### OONI presentation at the Internet Measurement Village 2020
On 10th June 2020, as part of the Internet Measurement Village 2020
(that we organized and hosted throughout June 2020), we live-streamed a
presentation about OONI.
The video recording of our presentation is available here:
https://www.youtube.com/watch?v=SES8PAeQvvs
### Online discussion of findings of OTF Information Controls Fellow
research report
Over the last year, OONI has served as the host organization for OTF
Information Controls Fellow, Chinmayi SK, whose research report was
published on the OONI website in mid-June 2020 (see:
https://ooni.org/post/2020-those-unspoken-thoughts-otf-fellow-report/).
On 25th June 2020, we hosted a session on the OONI Slack channel
(https://slack.ooni.org/) -- bridged with IRC -- to provide Chinmayi an
opportunity to present and discuss her findings with the community.
As part of this session, Chinmayi discussed her research findings in
depth, and received several questions from participants.
## Userbase
In June 2020, 7,657,872 OONI Probe measurements were collected from
5,942 networks in 208 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on “monthly coverage worldwide”):
https://explorer.ooni.org/
~ The OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200722/90481d03/attachment-0001.sig>
More information about the tor-project
mailing list