[ooni-dev] Available for collaboration - neumon.org

Wed May 6 13:13:47 UTC 2015

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/05/2015 16:19, balooni at espiv.net wrote:
> Hi Clodo,
>
> Thank you for your interest in OONI.
>
> Clodo wrote:
>
>> I'm the creator of the no-profit service http://www.neumon.org .
>> It's a project similar to OONI, but focused only on DNS and HTTP.
>
> Do you have the code published somewhere?
We release the source of the probe here:
https://github.com/AirVPN/neumon-probe
Written in C#/Mono. I run it from RaspBian on Raspberry PI.
But it's not a great piece of software. Simply it fetch from our backend
the list of domains to try to resolve/fetch, do it, and resend the
results. All detection are server-side based.

The backend it's written in php, sources never released. Contain
basically a lot of mysql queries to detect stuffs and generate report.
>
>> NeuMon browse collected DNS servers, and check if can be queried (open
>> and recursive). This because most of DNS ISP are recursive only from
>> it's customers subnet.
>> We maintain a huge list of domains to check (mix of known blocked
>> website, top alexa, etc).
>> Every DNS it's queried for each domains, we collect results, compare
>> against a known good value and discover custom injection (generally
>> that point to blocking page, i published some
>> example here: http://tinyurl.com/pl8znb4 ).
>>
>> So, i have:
>> - - a huge list of DNS servers, with country geolocation.
>> - - lists of domains blocked, country-based. Not exaustive.
>> - - i know many IP address that are destination of DNS redirection,
>> typically IP of servers that show html blocking pages. And DNS servers
>> of ISP that redirect to these addresses.
>
> It was quite difficult to find out and interpret the results from
> [1] could you maybe provide some pointers?
> 
Mainly because i don't publish all results, generally aggregated stats.

I have lists of domains blocked, but isn't available on neumon.org for
reason explained below in this mail.

http://www.neumon.org/blacklist.html
These IP are destination IP of DNS injection. We manage it manually.
They hosts the blocking pages. They have virtual-hosts in webserver for
domains, so you maybe cannot view the blocking page by viewing the IP
directly.
The lists contain also private services (like OpenDNS), not related to
censorship.
The lists may contain blocking page of private services (like
adult-filter services), not related to ISP censorship.
The recent tweet of Mikko Hypponen:
https://twitter.com/mikko/status/595681341334773760 are screenshots of
websites of the above IP list.

http://www.neumon.org/?view=dns_list&country=it
This is an example country list of DNS servers open and recursive we
detect in Italy.
Note that maybe include a customer of an ISP that have it's own DNS server.

Generally, i have a lot of data, catched automatically, that require
manual works to obtain nice and clean report, i'm in stall on this kind
of works.

>
>
> How did you find out about these domains and why do you think that they
> contain CP?
I don't know. Some of them (with domain name with keyword like
teen/sex/...) seem like typical porn website, a collection of video and
screenshot. Of course i can't know if are real CP.
I'm italian, and i know very well the italian situation: here ISP block
CP, gambling, proxy, file-sharing, file-hosting, webcam, pharma, escort,
drugs, steroid, etc.
Sometime, if a CP it's hosted on a public image-hosting, the entire
file-hosting services are blocked. ImageShack was DNS blocked for years
in Italy for a single CP image.
I obtain with my system lists of blocked domain, all of category listed
above together, but actually i don't want to publish it (see
https://youtu.be/RkmcupFx3FQ?t=1m13s ) because i can't detect CP versus
other categories.

Anyway, in my system sometime i have the information of what is
classified CP. For example, major ISP in Switzerland redirect CP domain
to a server hosted by stopp-kinderpornografie.ch .
>
>
> In any case it would be very interesting to see these results or
> of the ones that can be made public.
>
>> We also build a probe software, to allow other activists connected to
>> the ISP directly to launch it and detect censorship not based on DNS.
>
> It will be very interesting to instruct the probe software submit
> results to an ooni backend [2]. In any case the probe software can
> maybe even written as an ooniprobe test [3].
I understand you already have some DNS tests on ooniprobe. I will study
them.
But actually i don't understand what are the lists of domain tested by
OONI, how you detect spoof, and where/if you results are published.

My mysql data it's around 25 gb. I think maybe better (for maintenance
and independency) not to create OONI tests linked to neumon.org project.
I think maybe better if i create some webservices in neumon.org to
expose my data, where OONI backend can fetch interesting data for your
research.
For example, i can provide a list of DNS servers we detect (open to
query and with recursion enabled).
Or i can provide a list of
"open/recursive DNS Server IP -> query domain "xxx" -> the result "ip
address" it's probably a blocking page.

>
>> But nobody want to run a software that fetch also child pornography
>> domains, so nobody want to run our probe.
>
> I don't think that is all about CP only.
> Right now there are so many blacklists and censored websites worldwide
> and as far as I know people are interested in finding out of what
> resources are being blocked. Many of these started blocking
> gambling related websites and later added a bunch of other
> websites hence opening the door for censorship and blocking of other
> websites at will [4], [5].
A particular example: i know a very important ISP that redirect blocked
domain to a fixed IP.
Interesting and unbelivable, they specify the reverse-lookup info on
that IP.
So, a reverse lookup on that IP show thousand of domains, updated
frequently. I fetch periodically this list to populate my domains tests
list.

But, that list contain mixed CP, proxy, and in general all category of
blocked domains.
Detect what are the domains that may attract mainstream interest for
censorship reason, require filter that list by skipping CP, gambling
etc, and it's the kind of work that i don't know how to manage with an
automatic system.

I hope you can understand my poor english.

Ciao
Fabrizio - Clodo

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJVShOLAAoJEC/ixHrG0m4LZjoH/3ebg0oWuzEac4mNY51qBZ2C
MQm/apd0l+M+lYiFxJHY9xS7gmrAr/+YfudUieCfsLTuGAW0moeimU133f/pV+TJ
dbCGkUKRjVxKSp7D3ndijiSmFkOEtrNZrq2yEFZXYUAR2sSzz62VREa1XNqVDi6Z
hTKjAwwYDUesdD8DSO5UCPQu95s3gHwL4uk/28ExqmPddWu9cpoi3PD7gIhWjRJh
Q1wUESqCqnp6eVu4iS4UqUS6Gn/4I0rITOYNdgVZYXlYw0ENaPn8ILyHPEwLLLbY
DkWpPcrN+N1tRemAIBKlD5OHkBKSWFR8Qohyc0SUNnTz1qdDT/KS8YOyd6KdFqM=
=87nY
-----END PGP SIGNATURE-----