[tor-dev] Testing in Tor [was Re: Brainstorming a Tor censorship analysis tool]

[Nick Mathewson]

On Wed, Dec 19, 2012 at 8:57 PM, Simon <simonhf at gmail.com> wrote:
> On Wed, Dec 19, 2012 at 4:35 PM, Nick Mathewson <nickm at alum.mit.edu> wrote:
>> What's your favorite C mocking solution for integrating with existing
>> codebases without much disruption?
>
> This could be worth a separate thread. I'm not aware of really good
> solutions for C.
 [....]

I had a look around and found a few more possibilities, including:

  https://code.google.com/p/test-dept/
  http://throwtheswitch.org/white-papers/cmock-intro.html
  https://code.google.com/p/cmockery/

None of them looks compellingly great, TBH.  The methods that people
seem to be using code-rewriting tricks, mandatory macro tricks, LLVM
tricks, x86 assembly tricks, and uglier stuff still.

Perhaps somebody else has a good recommendation?  It would be sad if
we went ant built our own.

[...]
>> We're a part of the way there, then. Like I said, we've got multiple
>> network mocking/simulation tools.  With a simple Chutney network plus
>> the unit tests, we're at ~ 53% coverage... and all Chutney is doing
>> there is setting up a 10-node network and letting it all bootstrap,
>> without actually doing any end-to-end tests.
>
> Sounds good.
>
> I guess Chutney must be a separate project since I can't find it in
> the Tor sources .tar.gz ?

Yup.  It's accessible from gitweb.torproject.org.  I'd be surprised if
more than 5 people have tried to run it, ever.

(More results: unittests + chutney gives 52.60% coverage.  Unittests +
stem gives 39.03% coverage.  Unit tests + stem + chutney gives 54.49%
coverage.)

>> (ExperimenTor and Shadow are both heavier-weight alternatives for
>> running bigger networks, but I think that here they might not be
>> needed, since their focus seems to be on performance measurement.
>> Chutney is enough for basic integration testing, and has the advantage
>> that it's running unmodified Tor binaries.  Stem is interesting here
>> too, since it exercises Tor's control port protocol pretty heavily.)

More links:
https://shadow.cs.umn.edu/
http://crysp.uwaterloo.ca/software/exptor/

I'm not sure anybody's ever tried to do coverage with them.

 [..]
>> Yes, I like this idea a lot, especially if you're able to help with
>> it, especially if it's based on an already-existing
>> launch-a-network-on-localhost tool.
>
> I'm not aware of such a tool.

Chutney is such a tool; ExperimenTor can be made (I think) to act as
such a tool; Shadow is a little more complicated.

> The way I have done it in the past is to
> use Perl to lunch and monitor the various processes. The good thing
> about Perl is that it can run unmodified on both *nix and Windows,
> plus you can do one-liners.
[...]

Hm.  I'm not going to say that I'd turn down work in perl, but the
rest of the Tor developers don't spend much time using perl.  I don't
know that any of us have done a perl program of over 100 lines in the
last 5-6 years.  I'm not saying "perl sucks" or "I refuse to use
anything written in perl", but you should be aware that if you do
write anything in perl, there probably aren't a lot of other people
involved with Tor right now with the knowhow to effectively
collaborate on the perl parts or help to maintain them.

 [....]
>> I'm going to be travelling a lot
>> for the rest of December, but let's set up a time to chat in the new
>> year about how to get started.
>>
>> Preemptive Happy New Year,
>
> Dito. Sure let's set up a time.

[email sent off-list]

yrs,
-- 
Nick Mathewson