[tor-dev] Stem Integ tests for Tor

[Damian Johnson]

Hi Asis, glad you want to improve our testing of tor! That's one area
that could certainly use some love. ;)

> I would like to improve on Tor's codebase by writing Integ tests for it
> in Stem. I would like to know what kind of tests should be included
> in this. Also, atagar was talking about smoke tests for Tor as a whole.
> I would love to know everyone's ideas on the same.

We have two python frameworks for testing tor, chutney and stem.

The aim of chutney is to spawn a dedicated tor network for testing, so
we can make assertions based on the behaviour of multiple components.
For instance, 'if we make a one-hop circuit through an exit it should
be rejected'.

This is the proper solution for tor integration testing. The only
trouble is that doing this properly takes some work and chutney hasn't
been seeing much (any?) activity for quite some time.

The second test framework is stem. Its integration tests spawn a
*single* tor instance and exercise that, presently checking controller
functionality we can test while offline. Stem's integration tests are
a lot more mature than chutney, but since it spawns only a single tor
instance it's rather limited in the tor functionality it can test.

In the short term we can get a lot of mileage by expanding stem's
integration tests to exercise basic tor functionality. However, we'll
quickly hit a wall in terms of what we can test with stem. As such it
might be best for a GSoC project to start with a few stem integ tests
to boost its test coverage of tor, then switch over to improving
chutney.

One idea is...

1. Integrate gcov into stem's integration testing output.

2. Add some tests under the ONLINE target that cover basic functionality...

  * Construct a circuit to an exit which allows port 80 and request a
site through it.
  * Do the same for an exit which rejects port 80.
  * Construct a one-hop circuit through an exit.
  * Try to use a circuit that runs through a bridge.
  * Let tor pick the path of a circuit and assert that the first
relay's a guard and the last is an exit.
  * Attempt to request a page from a hidden service.
  * Host a hidden service and request something from ourselves.
  * Act as a relay and try to construct a circuit that includes ourselves.
  * ... etc...

I suspect those last two are no-gos because we would need to be
included within the consensus, but worth looking into.

Something to keep in mind is that the tor network is highly
unreliable. Tests should retry through alternate relays when they fail
to be sure that the failure is with our client rather than individual
relays.

3. Once we have a decently reliable set of tests to check basic tor
functionality move on to chutney. Nick would be able to best advise on
the work needed there.

> Also, I would like to start a new project which can automatically
> download & compile latest Tor. This would greatly help with the
> Integ tests for Stem & Tor.

Oops! My bad, I resolved the ticket for this but didn't update the
volunteer page (now fixed). We already have a jenkins instance that
does this...

https://jenkins.torproject.org/job/stem-tor-ci/

Cheers! -Damian