[tor-project] OONI Monthly Report: September 2019
Maria Xynou
maria at openobservatory.org
Wed Oct 2 16:03:37 UTC 2019
Hello,
Below we share what the OONI team worked on in September 2019.
## Launch of revamped OONI Explorer
We launched the revamped version of OONI Explorer!
Access the new OONI Explorer here: https://explorer.ooni.org/
Along with the launch, we published a post that explains in detail
what's new about OONI Explorer 2.0:
https://ooni.org/post/next-generation-ooni-explorer/
We were interviewed by the Committee to Protect Journalists (CPJ), who
published a Q&A where we discuss how journalists can use the new OONI
Explorer:
https://cpj.org/blog/2019/09/qa-ooni-explorer-team-unlock-the-webs-untold-censo.php
VPNCompare also published a blog post about the new OONI Explorer:
https://www.vpncompare.co.uk/ooni-explorer-uncovering-global-web-censorship/
## Report on blocking of BBC and Alhurra in Egypt
In collaboration with Egyptian community members, we published a
research report documenting the blocking of BBC and Alhurra in Egypt.
Our report is available here:
https://ooni.org/post/2019-egypt-blocks-bbc-and-alhurra/
Mohamed El-Taher translated the report to Arabic:
https://moeltaher.net/2019/09/26/مصر-تحجب-بي-بي-سي-والحرة-التوسع-في-الرق/
In the report, we share network measurement data on:
* Blocking of BBC:
https://explorer.ooni.org/search?until=2019-09-26&domain=www.bbc.com&probe_cc=EG
* Blocking of Alhurra:
https://explorer.ooni.org/search?until=2019-09-26&domain=www.alhurra.com&probe_cc=EG
We corroborated OONI measurements by performing manual curl tests with
the help of our local partners, the findings of which we share in the
report.
We found that Egyptian ISPs are using Deep Packet Inspection (DPI)
technology that is aware of TLS and which is most likely fingerprinting
the SNI field of the TLS handshake.
VPNCompare published a blog post covering our report:
https://www.vpncompare.co.uk/ooni-bbc-blocked-egypt/
## Updated test lists
To help improve the quality of measurements, we updated the following
test lists:
1. Algeria: https://github.com/citizenlab/test-lists/pull/517
2. Cameroon: https://github.com/citizenlab/test-lists/pull/508
3. Tanzania: https://github.com/citizenlab/test-lists/pull/520
4. Indonesia: https://github.com/citizenlab/test-lists/pull/512
5. Vietnam: https://github.com/citizenlab/test-lists/pull/516
As part of the process of updating the above test lists, we reviewed all
of the URLs (i.e. deleted defunct URLs, replaced old URLs with new ones,
added HTTPS where appropriate, etc.), fixed the category codes (i.e.
changed category codes where required), and added new URLs based on
relevant desktop research.
We also made contributions to other test lists:
https://github.com/citizenlab/test-lists/pull/514
https://github.com/citizenlab/test-lists/pull/511
https://github.com/citizenlab/test-lists/pull/510
https://github.com/citizenlab/test-lists/pull/509
https://github.com/citizenlab/test-lists/pull/506
https://github.com/citizenlab/test-lists/pull/502
https://github.com/citizenlab/test-lists/pull/501
https://github.com/citizenlab/test-lists/pull/500
## Progress on implementing new OONI Probe nettests
We made significant progress (80%) towards completing the implementation
of new OONI Probe nettests.
Specifically, we:
* Added functionality to measure SNI-based blocking:
https://github.com/ooni/netx/pull/11
* Completed the ndt7 client for OONI Probe:
https://github.com/m-lab/ndt7-client-go/pull/29
* Re-wrote DASH to use HTTPS: https://github.com/neubot/dash
* Extracted timing of network, TLS, HTTP events:
https://github.com/ooni/netx/blob/111e220d770f289108af03fc3ee5fdb526e4906a/internal/connx/connx.go#L12,
https://github.com/ooni/netx/blob/111e220d770f289108af03fc3ee5fdb526e4906a/internal/dialerapi/dialerapi.go#L167,
and
https://github.com/ooni/netx/blob/111e220d770f289108af03fc3ee5fdb526e4906a/internal/httptransport/httptransport.go
* Implemented DoT and DoH:
https://github.com/ooni/netx/tree/master/internal/dnstransport/dnsovertcp
and
https://github.com/ooni/netx/tree/master/internal/dnstransport/dnsoverhttps
* Got the content of DNS packets (including the DoT and DoH cases):
https://github.com/ooni/netx/blob/111e220d770f289108af03fc3ee5fdb526e4906a/internal/connx/connx.go#L70
## Progress towards stable release of OONI Probe desktop app for Windows
and macOS
We made progress on getting the OONI Probe desktop app ready for the
stable release.
Specifically, we:
* Completed the implementation of the Web Connectivity measurement
screen for the OONI Probe desktop app:
https://github.com/ooni/probe-desktop/pull/44
* Added support for dev tools to aid debugging:
https://github.com/ooni/probe-desktop/pull/43
* Wrapped strings (that are to be translated) in Formatted message to
support translations: https://github.com/ooni/probe-desktop/pull/41
* Upgraded the dependencies of the app to the latest version:
https://github.com/ooni/probe-desktop/pull/40
* Fixed some bugs related to upgrade depedencies:
https://github.com/ooni/probe-desktop/pull/39 &
https://github.com/ooni/probe-desktop/pull/42
The overall progress and next steps towards the stable release of the
OONI Probe desktop app can be tracked through our project board:
https://github.com/orgs/ooni/projects/4
## Progress towards stable release of OONI Probe Command Line Interface
(CLI)
As part of our work towards the stable release of the OONI Probe ClI, we:
* Prepared the v0.1.0 release of probe-engine:
https://github.com/ooni/probe-engine/pull/62
* Documented the milestones for the stable probe-cli:
https://github.com/ooni/probe-cli/milestone/3
## Progress on OONI Probe mobile apps
We continued to make progress towards the public release of OONI Probe
mobile v.2.2.0.
Android:
https://github.com/ooni/probe-android/pull/263
https://github.com/ooni/probe-android/pull/262
https://github.com/ooni/probe-android/pull/258
https://github.com/ooni/probe-android/pull/256
iOS:
https://github.com/ooni/probe-ios/pull/308
https://github.com/ooni/probe-ios/pull/307
## Progress towards reducing the latency of publication of measurements
We continued to make progress towards reducing the latency of the
publication of uploaded measurements. This can be viewed through our
work on the fast-path: https://github.com/ooni/pipeline/pull/213
The fast-path code has been reviewed and it is currently deployed
against active OONI Probe collectors.
The remaining next step is to add support for returning the realtime
measurements via OONI API queries, and to support the searching and
listing of these measurements in OONI Explorer. This is work in
progress: https://github.com/ooni/api/pull/115
## Progress on data processing pipeline heuristics for flagging relevant
blocking events for the RSS feed
To flag relevant blocking events via the RSS feed in OONI Explorer, we
created a new data pipeline with an internal web app and API for the
publication of charts and we worked on integrating the blocking events
into the database.
The component implements a new heuristic to analyze past events and
infer when URLs become blocked or unblocked with granularity of a country.
Current branch and pull request: https://github.com/ooni/pipeline/pull/228
We expect to have the event engine running soon and we'll then work on
publishing it on OONI Explorer.
## Ensuring that our infrastructure is re-deployable
As part of our efforts to ensure that all our infrastructure is
re-deployable, we made progress on the following:
* The dom0 bootstrap script has been updated to work with Debian buster:
https://github.com/ooni/sysadmin/pull/353
* We dropped the legacy UID scheme to make it easier to deploy machines:
https://github.com/ooni/sysadmin/issues/289
* We upgraded matomo to a latest machine, redeploying it:
https://github.com/ooni/sysadmin/pull/355 (fixes:
https://github.com/ooni/sysadmin/issues/225 &
https://github.com/ooni/sysadmin/issues/149)
* Grafana was updated to the latest version:
https://github.com/ooni/sysadmin/pull/360
The master ticket with all the hosts that need to be re-deployed is
available here: https://github.com/ooni/sysadmin/issues/356
## Improving the availability of our infrastructure
We made progress on improving the availability of our infrastructure.
Our plan on how to go forward has been documented in this ticket:
https://github.com/ooni/sysadmin/issues/359
Our work on oonifying the probe services is available in this pull
request: https://github.com/ooni/sysadmin/pull/361
So far, we have deployed a new web-connectivity test helper and we will
oonify the bouncer, collector, and orchestra backends into the same host.
## Community use of OONI data
### OONI data supports court case in Pakistan
Back in November 2017, we reported on the blocking of several media
websites, Facebook (facebook.com and Facebook Messenger), YouTube,
Twitter, Instagram, WhatsApp, and Telegram in Pakistan amid protests:
https://ooni.io/post/how-pakistan-blocked-social-media/
Our report -- which shared OONI network measurement data collected from
Pakistan -- was used as evidence in a petition filed in the Islamabad
High Court by Pakistan's Media Matters for Democracy
(http://mediamatters.pk/), challenging the blocking of media websites
and popular social media platforms during the protests.
The petition challenged the Pakistan Telecommunication Authority's (PTA)
authority to block websites without due process. Details are available
here:
http://digitalrightsmonitor.pk/islamabad-high-court-takes-up-petition-filed-by-rights-activists-challenging-the-legality-of-social-media-takedown-during-faizabad-dharna-issues-notices-to-pta-and-the-federal-government/
The petition was filed through Charahgar
(http://mediamatters.pk/charahgar/), a legal aid centre that Media
Matters for Democracy established with the support of Media Legal
Defense Initiative (MLDI).
As an outcome, the Islamabad High Court declared (in September 2019)
that the Pakistan Telecommunications Authority's interpretation of
Section 37 of PECA is against the Constitution, and that they cannot
regulate content without following the due process, the principles of
transparency, and oversight. Details are available here:
https://tribune.com.pk/story/2061933/1-pta-not-empowered-block-website-ihc/
This is a landmark victory for internet freedom in Pakistan.
### Blocking of change.org in Cuba
Through the use of OONI Probe and OONI data, YucaByte (a Cuban
non-profit organization) published a report on the blocking of
change.org in Cuba:
http://www.yucabyte.org/confirman-bloqueo-changeorg-en-cuba/
Their report (including the use of OONI Probe) received the following
press coverage:
https://adncuba.com/noticias-de-cuba/tecnologia/confirman-bloqueo-de-la-plataforma-changeorg-en-cuba
### Venezuelan partners nominated for 2019 Gabo Award
Through the use of OONI Probe and OONI data, IPYS Venezuela (along with
Venezuela Inteligente / VEsinFiltro) has been monitoring internet
censorship in Venezuela since 2014. They have created a network of
volunteer OONI Probe users across Venezuela, who coordinate on measuring
and reporting on emergent censorship events. Through the use of OONI
data, they have published numerous research reports, documenting
internet censorship events in Venezuela on an ongoing basis.
Their Intercortados project (https://ipysvenezuela.org/intercortados/)
has been nominated for the 2019 Gabo Award, among the 10 best innovation
projects.
Further information:
https://premioggm.org/premio-gabo/edicion/2019/nominados/
https://premioggm.org/noticias/2019/09/conoce-a-los-40-nominados-al-premio-gabo-2019/
https://twitter.com/Mariengracia/status/1172198403478040580
## Community activities
### Community meeting
We held our monthly community meeting on Slack
(https://slack.openobservatory.org/) on Tuesday, 24th September 2019.
During the meeting, we discussed:
1. Launch of revamped OONI Explorer: Update and questions
2. Query speed problem in API
3. OONI Probe testing in Egypt
## Userbase
In September 2019, OONI Probe was run 417,684 times from 5,407 different
vantage points in 218 countries around the world.
This information can also be found through our stats:
https://api.ooni.io/stats
~ The OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.torproject.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20191002/ed931800/attachment.sig>
More information about the tor-project
mailing list