[tor-dev] Testing in Tor [was Re: Brainstorming a Tor censorship analysis tool]

[Nick Mathewson]

On Wed, Dec 19, 2012 at 2:29 PM, Simon <simonhf at gmail.com> wrote:
 [...]
> Maybe there is no automated testing for any Tor projects? At least a
> quick search on the wiki only found [1] which lists possible ways to
> test (but was created 7 months ago and apparently not updated since
> and collecting dust) and [2] discussing a manual test procedure for
> TBB. However, tor-0.2.3.25.tar.gz does reveal some test files but the
> source code ratio of production code to test code is not inspiring at
> first glance:
[...]

Be aware that we've also been using 'chutney' and 'experimentor' for
integration testing.  They supplement coverage a bit, though they need
more tests, and each tends to hide certain classes of error.

> Tor seems to have good planning compared to most open source projects.
> So I would be interested in hearing why testing is apparently 'falling
> between the cracks'. Why isn't there just 10 times more test LOC?

Not because of any hatred or disapproval of tests--just because
nobody's written that 100 kloc of testing code yet.

I think that's for two main reasons:
  * We were in a hurry when we wrote lots of the stuff we wrote.
  * Large parts of the codebase have been written in a tightly coupled
style that needs refactoring before it can be tested without a live
Tor network at hand.
  * Until Tor 0.2.2, our testing framework didn't let us have tests
that touched global state, which made our tests in general pretty
fragile.

>   What
> about implementing a new policy immediately: Any new production LOC
> committed must be covered by tests, or peer reviewed and
> democratically excluded?

Goodness knows we need more review and testing.

It doesn't seem fair to reject patches for lacking test coverage when
they are patches to code that itself lacks coverage, though.  If you
write a 5-line patch to connection_ap_rewrite_and_attach, for
instance, you probably shouldn't be on the hook for refactoring the
whole function to make it testable, though you will be hard-pressed to
write any tests for that monster without a complete refactor.

It might be a reasonable goal to try to set a plan for increasing test
coverage by a certain percentage with each release.

If you like and you have time, it would be cool to stop by the tickets
on trac.torproject.org for milestone "Tor: 0.2.4.x-final" in state
"needs_review" and look to see whether you think any of them have code
that would be amenable to new tests, or to look through currently
untested functions and try to figure out how to make more of them
tested and testable.

yrs,
--
Nick