[tor-project] OONI Monthly Report: November 2019

[Maria Xynou]

Hello Tor world,

Happy new year!

In this email I share OONI's status report for November 2019, and our
December 2019 report will follow next week. Apologies for the delay!

# OONI Monthly Report: November 2019

In November 2019, OONI started working on the DRL Anti-censorship grant
in collaboration with The Tor Project.

We also started using 2 week cycles to track progress, effort, and
priority estimation in order to improve how we estimate and track the
work we do on projects.

In particular, the work related to the DRL Anti-censorship grant is
being tracked on the following public GitHub project board:
https://github.com/orgs/ooni/projects/18

## OONI measurements are now near real-time

We are very excited to announce that starting from November 2019, OONI
measurements are getting published on OONI Explorer and in the API in
near real-time!

This means that from the moment an OONI Probe user runs a test to it
appearing in OONI Explorer, the delay has gone down from a maximum of
24h to an order of minutes.

## Report on internet blackout in Iran

In collaboration with IODA and Iranian community members (Kandoo), we
co-published a research report examining the internet blackout in Iran:
https://ooni.org/post/2019-iran-internet-blackout/

Community members translated the report to Persian:
https://paskoocheh.com/blog/posts/2019-11-29-iran-internet-shutdown-ooni-report.html

## Psiphon test integration

The primary focus during November was the integration of the Psiphon
test into probe-engine, probe-cli and consequently the OONI Probe
Desktop app.

We picked this as the first item as we had already done a bit of cursory
research into the problem and had a clear idea of how we would go about
integrating and supporting it.

The integration work done for the Psiphon test is also very propaedeutic
towards doing other golang based circumvention tests, such as obfs4proxy
and other golang based Tor pluggable transports.

Our goal is that of getting Psiphon integrated inside of probe-engine,
so that we are able to run the test from probe-cli which is in turn run
by the OONI Probe Desktop app.

This issue is tracked as part of:
https://github.com/ooni/probe-engine/issues/89

Throughout November 2019, we made a lot of progress and completed most
of the items necessary to achieve this goal.

In particular, we ensured we’re using the latest version of Psiphon
dependencies (see: https://github.com/ooni/probe-engine/issues/141),
made sure the data format of the test result matches the old
implementation (see: https://github.com/ooni/probe-engine/issues/128)
and re-enabled the ability to build the Psiphon test inside of probe-cli
(see: https://github.com/ooni/probe-cli/issues/79), making it possible
to build the Psiphon test from the command line interface used by the
OONI Probe desktop app.

Other completed issues relevant to Psiphon integration include:

* httpx implement NewClientWithProxy
(https://github.com/ooni/netx/issues/130), which allows us to test that
Psiphon is working by using it
* HTTPDoConfig: add support for proxy function
(https://github.com/ooni/netx/issues/131), which enables using Psiphon
as a proxy
* porcelain: add proxy and byte counting support
(https://github.com/ooni/netx/pull/134), which allows us to measure the
amount of bytes sent and received over the Psiphon proxy
* x/logger: mimic cURL logging
(https://github.com/ooni/netx/issues/137), which has been useful for
more easily debugging the new Psiphon test
* psiphon: use netx so we fill the HTTP template
(https://github.com/ooni/probe-engine/issues/129), which allows us to
collect low-level events and generate the Psiphon test data format
* orchestra: implement authenticated requests
(https://github.com/ooni/probe-engine/issues/148), which allows us to
fetch the Psiphon configuration from the OONI orchestra services

On the OONI Probe desktop user interface front, with support from
Antonela from the Tor UX team, we designed and implemented the
Circumvention test card which will allow us to trigger the Psiphon test,
but also all the circumvention tests we will be developing.

For reference, see:
* Add circumvention tool tests icon:
https://github.com/ooni/design-system/issues/72
* Mockups for the Performance test card:
https://github.com/ooni/probe/issues/804#issuecomment-559824046

Based on these mockups, we worked on adding support for triggering the
circumvention tests inside of the OONI Probe Desktop app, see:
https://github.com/ooni/probe-desktop/pull/88.

## OONI testing engine improvements

We made a series of improvements related to the OONI Probe testing
engine such as adding support for serialising binary header values in
probe-engine (https://github.com/ooni/probe-engine/issues/110) and
adding documentation for it in ooni/spec
(https://github.com/ooni/spec/issues/164).

When performing HTTP requests in the Telegram golang test rewrite, we
are now sure to include the Accept and Accept-Language headers to look
more similar to a real browser
https://github.com/ooni/probe-engine/issues/113.

We also dealt with several other low hanging fruit in the probe-engine
and ooni-spec.

See:
* porcelain: confusing that two errors are returned   
https://github.com/ooni/netx/issues/141
* dash: write tests and reach 100% coverage   
https://github.com/ooni/probe-engine/issues/93
* Bump data format to v0.3.0: https://github.com/ooni/spec/issues/163
* Bump data format to v0.3.0:
https://github.com/ooni/probe-engine/issues/111
* Update to ooni/probe-assets at 20191129161951:
https://github.com/ooni/probe-engine/issues/116
* Update well know OONI hosts we use:
https://github.com/ooni/probe-engine/pull/117

### OONI Probe Desktop app

Beyond the Psiphon test specific improvements listed above, we also made
several smaller improvements to the OONI Probe Desktop app, mostly
related to UX and App stability.

We added a link to the risks documentation in the onboarding screen
(https://github.com/ooni/probe-desktop/issues/60), fixed a UI issue with
some icons not being aligned
(https://github.com/ooni/probe-desktop/issues/71) and made the macOS dmg
disk image more pretty (https://github.com/ooni/probe-desktop/issues/31).

We also did some brainstorming on how we could improve the usability of
the card descriptions in the home page of the desktop app
(https://github.com/ooni/probe-desktop/issues/63 &
https://github.com/ooni/probe-desktop/issues/91).

### OONI Explorer and OONI API

On the OONI Explorer front, our focus was on improving the application
performancing by dealing with low hanging fruit we are sure is going to
make improvements to it, while at the same time collecting more metrics
to evaluate where we should be directing our other efforts related to
app performance.

In OONI Explorer, we fixed an issue with the OONI API requests being
launched twice during search operations:
https://github.com/ooni/explorer/issues/350.

## Infrastructure and backend improvements

During this month we dealt with several improvements and incidents
affecting our backend infrastructure, such as:

* An incident related to the collector running out of disk space:
https://github.com/ooni/pipeline/issues/265.
* We implemented a pattern that allows us to deploy application using
debian packages: https://github.com/ooni/pipeline/pull/264
* We resolved an issue with the input field missing from the
fastpath:    https://github.com/ooni/pipeline/pull/255
* We merged and deployed the event detector in production:
https://github.com/ooni/pipeline/pull/228

## Test list updates

We collaborated with OutRight International on updating multiple test
lists to include (more) LGBTQI websites:
https://github.com/citizenlab/test-lists/pull/539

We also reviewed several test list updates by Netalitica researchers and
opened pull requests for the following test lists:

* Pakistan: https://github.com/citizenlab/test-lists/pull/541
* Tunisia: https://github.com/citizenlab/test-lists/pull/543
* Sudan: https://github.com/citizenlab/test-lists/pull/544
* United Arab Emirates: https://github.com/citizenlab/test-lists/pull/545

## Community use of OONI data

### Freedom on the Net 2019 reports

OONI data/reports were cited in the following Freedom on the Net 2019
reports:
   
* Venezuela:
https://www.freedomonthenet.org/country/venezuela/freedom-on-the-net/2019

* Zimbabwe:
https://www.freedomonthenet.org/country/zimbabwe/freedom-on-the-net/2019

* Zambia:
https://www.freedomonthenet.org/country/zambia/freedom-on-the-net/2019

* Nigeria:
https://www.freedomonthenet.org/country/nigeria/freedom-on-the-net/2019

* Kenya:
https://www.freedomonthenet.org/country/kenya/freedom-on-the-net/2019

* Jordan:
https://www.freedomonthenet.org/country/jordan/freedom-on-the-net/2019

* Iran: https://www.freedomonthenet.org/country/iran/freedom-on-the-net/2019

* Indonesia:
https://www.freedomonthenet.org/country/indonesia/freedom-on-the-net/2019

* Ethiopia:
https://www.freedomonthenet.org/country/ethiopia/freedom-on-the-net/2019

* Egypt:
https://www.freedomonthenet.org/country/egypt/freedom-on-the-net/2019

* Cuba: https://www.freedomonthenet.org/country/cuba/freedom-on-the-net/2019

OONI data is also indirectly cited via the following reports (which cite
the reports of community members that use OONI data):
   
* Ukraine:
https://www.freedomonthenet.org/country/ukraine/freedom-on-the-net/2019

* Malaysia:
https://www.freedomonthenet.org/country/malaysia/freedom-on-the-net/2019

## Community activities

### OTF Summit

OONI's Arturo and Maria traveled to Taiwan to attend the OTF Summit (4th
& 5th November 2019) where they facilitated OONI workshops. They also
participated in the Open Taiwan Day on 6th November 2019.

### Blog post by Karisma Foundation

Our Colombian partners, Karisma Foundation, published a blog post where
they encourage the use of OONI Probe during the November 21st strike:
https://stats.karisma.org.co/ayudanos-a-establecer-y-documentar-posibles-bloqueos-a-medios-durante-el-paro-del-21-de-noviembre/

### Community meeting

We facilitated the monthly OONI Community Meeting on Tuesday, 26th
November 2019, on the OONI Slack channel (https://slack.ooni.org/).

As part of the meeting, we discussed:
   
1. Comprehensive testing of protocols

2. Testing the new OONI Probe Desktop apps:
https://github.com/ooni/probe-desktop/releases/tag/v3.0.0-rc.3

3. Measuring internet blackouts from the edge of the network

4. Participating at the IGF

5. Request for feedback on OONI website re-design, FAQ, and glossary

## Userbase

In November 2019, OONI Probe was run 406,846 times from 5,417 different
vantage points in 205 countries around the world.

This information can also be found through our stats on the OONI
Explorer landing page: https://explorer.ooni.org/

~ The OONI team

-- 
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.torproject.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200103/1e3be45f/attachment.sig>