[tor-project] OONI Monthly Report: January 2020
Maria Xynou
maria at openobservatory.org
Wed Feb 5 11:41:51 UTC 2020
Hello,
Happy new year, and happy new decade!
## OONI is an independent entity
As of 1st January 2020, OONI is an independent entity!
Back in 2011, OONI was born out of the Tor Project, which has since
served as our fiscal sponsor. Thanks to Tor Project support, OONI was
able to grow and become what it is today. We are immensely grateful to
the Tor Project for all their support over the last 8 years, and we look
forward to continuing to collaborate on the fight against internet
censorship!
As part of our growth, OONI is transitioning to greater autonomy and
independence. As of 1st January 2020, OONI is fiscally sponsored by the
Hermes Center for Transparency and Digital Human Rights
(https://www.hermescenter.org/), a non-profit, digital rights
organization based in Italy, which was co-founded by Arturo Filastò
(OONI’s co-founder) back in 2011. OONI’s Maria Xynou and Simone Basso
are also members of the Hermes Center. We may eventually create a
dedicated-to-OONI NGO, but for now, the current arrangement is sufficient.
## New project management methods
As of December 2019, we have changed how we do project management in our
team.
We now work in 2-week sprint cycles, based on which we plan and track
the progress of our activities. So far, this works really well!
We are using Zenhub boards to plan and estimate our activities, all of
which are public, but require registration with the app.
Here is our Sprint Planning board:
https://app.zenhub.com/workspaces/sprint-planning-5dfa4b7355438f7e86c80e5a/board?repos=8707973,46408517,118942953,20287787,10014717,83037093,3944100,187713647,101668376
Even if the internal planning is behind a registration wall, all of our
activities and priorities are still tracked using native GitHub tooling,
making it possible for anyone to see what our priorities are via the
public GitHub interface.
We have also consolidated our GitHub issue repositories to only 9 in total:
* https://github.com/ooni/ooni.org
* https://github.com/ooni/backend
* https://github.com/ooni/probe
* https://github.com/ooni/probe-engine
* https://github.com/ooni/explorer
* https://github.com/ooni/run
* https://github.com/ooni/design-system
* https://github.com/ooni/orchestra
This ticket shares more details: https://github.com/ooni/ooni.org/issues/317
To see what we are working on over the next 2 weeks for a given project,
just select the next milestone of that repository on GitHub.
Each issue should also have a priority label assigned to it and an
effort estimate label.
Each OONI team sprint has a marine-inspired name, and we encourage
community members to propose names for our sprints here:
https://github.com/ooni/ooni.org/issues/337
## OONI Probe support for running circumvention tool tests
During January 2020, we almost completed the implementation of all
client-side logic that is required for testing Psiphon and Tor inside of
OONI Probe.
To add support for running Tor, we:
* Wrote a first iteration of the Tor test:
https://github.com/ooni/probe-engine/issues/226
* Added support for the Tor bridge reachability test in probe-engine:
https://github.com/ooni/probe-engine/issues/2
* Added support for running obfs4proxy in probe-engine:
https://github.com/ooni/probe-engine/issues/90
* Designed how the Tor test results should be presented in the OONI
Probe desktop app: https://github.com/ooni/probe/issues/967
* Made a release of probe-cli with support for the Tor test experiment:
https://github.com/ooni/probe/issues/974
* Exposed top level keys to expose top level statistics in the UI:
https://github.com/ooni/probe-engine/issues/292
* Almost completed the UI work required to show the Tor test results in
the OONI Probe app: https://github.com/ooni/probe/issues/975
To add support for running Psiphon, we:
* Finalized support for running Psiphon in probe-engine:
https://github.com/ooni/probe-engine/issues/89
* Finished integrating the Psiphon test results with the correct Psiphon
icon in the OONI Probe Desktop App
* Designed the UI for the circumvention tests in general:
https://github.com/ooni/probe/issues/804
## OONI Probe orchestration logic for circumvention tool testing
In order to support measuring Psiphon and Tor, OONI Probe clients need
to know which configurations they should use to carry out the testing.
We therefore added support for giving out such testing targets (Psiphon
configuration, Tor default bridges, Tor directory authorities) via
authenticated backend calls.
Specifically, we:
* Added support for giving out Psiphon configurations:
https://github.com/ooni/orchestra/issues/78
* Added support for giving out addresses for Tor tests:
https://github.com/ooni/orchestra/issues/82
## Making OONI Probe reporting logic more resilient to censorship
We made a bit of progress in understanding how we can make OONI Probe
testing more resilient to blocking and we added extra robustness to the
app to overcome potential blocking attempts.
Specifically, we:
* Improved how we display measurements which have failed due to a
bouncer failure: https://github.com/ooni/probe/issues/855
* Discussed solutions for what we should do when the IP lookup of a
measurement fails: https://github.com/ooni/probe/issues/879
* Investigated how to allow alternative methods of sending measurement
results to us via email (https://github.com/ooni/probe/issues/992) or IM
apps (https://github.com/ooni/probe/issues/976)
## Analysis of circumvention tool test data and presentation in OONI
Explorer
We made a lot of progress with respect to presenting the new Psiphon and
Tor test results inside of OONI Explorer and exposing them via the OONI
API. We also worked on the scoring and analysis of the results in the
data processing pipeline.
Specifically, we:
* Added support in the batch pipeline for scoring Tor test results:
https://github.com/ooni/backend/issues/293
* Wrote data processing logic for processing Psiphon test results:
https://github.com/ooni/backend/issues/209
* Re-wrote the OONI API list_measurements API call, adding support for
listing and searching for Psiphon and the new Tor test results and
boosting its performance: https://github.com/ooni/backend/issues/160
* Almost finished adding support for presenting Psiphon test results in
OONI Explorer: https://github.com/ooni/explorer/issues/332
* Made considerable progress on presenting Tor test results in OONI
Explorer: https://github.com/ooni/explorer/issues/332
To better tune the analysis engine, we need to ship the tests and
increase the measurement coverage.
## OONI data processing pipeline
In November 2019, we started publishing OONI measurements from around
the world in near real-time (i.e. within minutes). This, however, had an
impact on the performance of the OONI API (which OONI Explorer relies on).
To boost the performance of the OONI API (and, by extension, the
performance of OONI Explorer), we made a lot of progress on the OONI
data processing pipeline, all of which can be viewed through the
following tickets:
https://github.com/ooni/backend/issues/161
https://github.com/ooni/backend/issues/163
https://github.com/ooni/backend/issues/160
Throughout January 2020, we added many blockpage fingerprints to the
fast-path pipeline, making it possible to automatically confirm many
more cases of internet censorship. See:
https://github.com/ooni/pipeline/pull/289
We also experimented with adding support for calculating some statistics
of measurements and anomalies. See:
https://github.com/ooni/backend/issues/292
Moreover, we added support in the fast-path pipeline for anomaly,
failure, and confirmed columns. We also worked on adding support for
reconnections during failing SSH conditions, as we had noticed issues
with this.
We added support for ingesting the Citizen Lab category codes for URLs,
in order to eventually support filtering URLs by category on OONI
Explorer. This work is available here:
https://github.com/ooni/backend/issues/231
We also discussed how we are going to move the development of the OONI
data processing pipeline forward and came up with a plan for that, as
documented in the following tickets:
https://github.com/ooni/backend/issues/195
https://github.com/ooni/backend/issues/286
## OONI Probe mobile app
During January 2020, we added support for setting no limit to the
maximum runtime.
It is now possible for OONI Probe mobile app users to set the Websites
test max_runtime to zero in order to test entire test lists.
See: https://github.com/ooni/probe/issues/893
We also made several important bug fixes related to how certain error
conditions are being handled. See:
https://github.com/ooni/probe/issues/865 &
https://github.com/ooni/probe/issues/926
## OONI Probe measurement engine
We introduced timing and TLS in the testing (and other low-level data
collection), to ensure that OONI Probe tests are as useful as possible.
Essentially, OONI Probe tests that can make use of this low-level
information do so. This is an important enhancement (and an essential
building block) to the Tor test and other tests.
## Data analysis
We analyzed OONI measurements for the following:
* Azerbaijan Internet Watch (partner):
https://github.com/ooni/ooni.org/issues/332
* OTF Information Controls Fellow (hosted under OONI):
https://github.com/ooni/ooni.org/issues/331
* Research on the blocking of LGBTQI sites worldwide:
https://github.com/ooni/backend/issues/224
## OONI backend
We only had one major new incident to deal with in January 2020, which
is documented here: https://github.com/ooni/sysadmin/issues/422
We had a relapse of an old incident
(https://github.com/ooni/sysadmin/issues/128) and we worked on coming up
with a fix to this here: https://github.com/ooni/backend/issues/300
## Published OONI FAQ
We published a Frequently Asked Questions (FAQ) section on our website:
https://ooni.org/support/faq/
Through the OONI FAQ, we aim to address the most frequently asked
questions that we have received from community members over the last years.
The questions fall under the following areas:
* About OONI
* OONI Probe
* Testing websites
* OONI data
* OONI Explorer
We aim to continue to update the OONI FAQ on an ongoing basis, and we
encourage further community feedback.
## Published OONI Glossary
We published an OONI Glossary on our website:
https://ooni.org/support/glossary
This glossary is meant to cover terms that are commonly used in the
OONI-verse, within OONI apps, documentation, and research reports. As a
result, this glossary is quite OONI-specific.
That said, it also includes a variety of terms that are commonly used by
many other projects and may therefore be useful to others in the broader
internet freedom community.
Similarly to the OONI FAQ, we aim to continue to improve upon and update
the OONI Glossary on an ongoing basis. We encourage further community
feedback, and suggestions can be shared directly on GitHub:
https://github.com/ooni/ooni.org
## Collaboration with Netalitica
We reviewed the test list updates by Netalitica researchers and opened
pull requests for the following test lists:
* Turkmenistan: https://github.com/citizenlab/test-lists/pull/552
* Uzbekistan: https://github.com/citizenlab/test-lists/pull/553
* Jordan: https://github.com/citizenlab/test-lists/pull/554
* China: https://github.com/citizenlab/test-lists/pull/555
We also reviewed Netalitica updates to the Russian
(https://github.com/citizenlab/test-lists/pull/557), Turkish and Iranian
(https://github.com/citizenlab/test-lists/pull/564) test lists.
## Organization of IFF Internet Measurement Village
This year OONI is co-organizing an Internet Measurement Village during
the last 2 days of the Internet Freedom Festival (IFF) in Valencia, Spain.
We reached out to several other measurement projects, inviting them to
co-host the Internet Measurement Village with us.
Throughout January 2020, we coordinated with the IFF organizers and with
the other Internet Measurement Village members on logistics, and we
organized and hosted two brainstorming calls. We also collected session
ideas from community members, based on which we are creating the agenda
for the Internet Measurement Village.
## Organization of OONI Partner Gathering
In July 2017, we organized and hosted the first OONI Partner Gathering:
a two-day event that brought all of our partners from around the world
together to share skills and knowledge around censorship measurement.
We plan to host the next OONI Partner Gathering in 2020! To this end, we
wrote and submitted funding proposals, organized relevant logistics, and
worked on updating relevant budgets.
## Community activities
### OONI presentation at the Nexa Center
On 22nd January 2020, Arturo presented OONI and discussed censorship
findings from 2019 at the Nexa Center for Internet and Society in Turin,
Italy.
Information about his presentation is available here:
https://nexa.polito.it/lunch-75
### Community meeting
We facilitated the monthly OONI Community Meeting on 28th January 2020
on our Slack channel (https://slack.ooni.org/), during which we discussed:
1. Updates from the OONI team
2. Setting up a proxy for OONI Probe Android and iOS
3. Maxmind licensing problems
## Userbase
In January 2020, OONI Probe was run 8,142,429 times from 5,307 different
vantage points in 206 countries around the world.
This information can also be found through our measurement stats on OONI
Explorer (see chart on monthly coverage worldwide):
http://explorer.ooni.org/
~ The OONI team.
--
Maria Xynou
Research & Partnerships Director
Open Observatory of Network Interference (OONI)
https://ooni.org/
PGP Key Fingerprint: 2DC8 AFB6 CA11 B552 1081 FBDE 2131 B3BE 70CA 417E
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-project/attachments/20200205/abfd211b/attachment-0001.sig>
More information about the tor-project
mailing list