[tor-dev] Brainstorming a Tor censorship analysis tool

[Arturo Filastò]

On Dec 18, 2012, at 8:07 PM, Philipp Winter <identity.function at gmail.com> wrote:

> Hi there,
> 
> Deliverable 6 for sponsor Z says:
> 
>> 6. Start a tool that a censored developer can run to discover why their Tor is
>> failing to connect: brainstorm a list of "things to check", and sort them by
>> how useful they'd be to check / how hard they'd be to build. (#7137)
> 
> The deliverable is due on Feb. 28, 2013 so we should get started.
> 
> Some background about the deliverable:
> The reason for this project is that debugging possible censorship events is
> tedious right now. We often have no access to machines in censoring countries
> and we are dependent on users creating packet dumps for us. This tool should
> speed up and automate this process to some extent. Censored users should run it
> and the tool should then collect data which should then somehow reach us.
> 
> I created the following wiki page which should contain all the necessary
> information:
> https://censorshipwiki.torproject.org/TorCensorshipAnalyzer
> 
> Please add/modify stuff and share your opinion. Since there is quite some
> overlap with OONI, it would be great if the OONI people could give feedback.
> 

I believe you should be using ooniprobe to build a the tests you are interested
in building, or you may at least be interested in looking at our code to see 
how to do the things you are interested in doing.

The main points where ooniprobe would be of use to you (now) are:

# Standard reporting format

All ooniprobe tests share a common base format depending on the test template 
your test is based on.

I recommend you look at the Test Writing tutorial to get an idea of how this
looks like:
https://ooni.torproject.org/docs/writing_tests.html

# Collection of packet captures

When you run an ooniprobe test and you have set your ooniprobe.conf file to 
"includepcap: true" then you will collect a full pcap of what has happened on 
the probes network during the test run.

Note: This requires the test to be run as root and will include *all* the network
traffic during the testing session (i.e. if the user is looking at their favorite 
kitten website while running the test, such data will be in the pcap)

# Collection of packet captures specific to the sent and received packets

When you run a ooniprobe test that inherits from the scapy test template (https://ooni.torproject.org/docs/api/ooni.templates.html#module-ooni.templates.scapyt) 
the packets sent and received (i.e. that are answers to the packet(s) sent) will be 
captured.

When configured to not include the probe IP address, source IP of sent packets 
and dst IP of received packets is replaced with 127.0.0.1. (warning: if the IP 
address of the probe is present in some other parts of the packet it will not get 
stripped, for example if it's present in the ICMP citation)

# Reporting system

Currently we only support collection of YAML formatted reports (that means not 
.pcap files) and only via Tor Hidden Services.

Extending it to support reporting via HTTP(s) should be trivial and is a feature 
that we have already received a request for.

Adding support for collecting also .pcaps also probably does not require that much 
amount of time and is something that will happen in the near future.

# Things to come

ooniprobe will soon expose a HTTP based API that binds to localhost that can then
be (optionally) exposed as a Tor Hidden Service. Such API will allow researchers to
connect to a probe and run some tests and will allow us to build a JS/HTML5 client
interface to allow users to select which tests to run and monitor the status of running
tests.

More details here:
https://ooni.torproject.org/docs/architecture.html#ooniprobe-api

For a birds-eye view of the project see:
https://ooni.torproject.org/docs/architecture.html

Even if you do not end up using ooniprobe for developing your system today, I 
highly encourage you to use the libraries that we are using so that in the 
future we can find a way to integrate code from each others projects.

The main libraries that we are using are:

* Twisted http://twistedmatrix.com
* Scapy http://www.secdev.org/projects/scapy/
* txtorcon https://github.com/meejah/txtorcon


~ Art.