[tor-dev] Unit testing blog posts

[Esteban Manchado Velázquez]

Hey Damian!

Wasn't sure if follow up privately or to the list, sending this to the  
list for now.

On Thu, 16 Feb 2012 22:09:44 +0100, Damian Johnson <atagar1 at gmail.com>  
wrote:

> Nice posts and thanks for improving the tor tests! I'm not entirely in
> agreement with the last point (about tests covering all cases). If the
> test space is decently small then exercising everything can better
> ensure that you don't violate a set of invariants. For example, one of
> stem's unit tests attempt every combination of authentication methods
> against every set of failures they can encounter, making sure that we
> properly report success/failure and never raise an unexpected type of
> exception...
> https://gitweb.torproject.org/stem.git/blob/HEAD:/test/unit/connection/authentication.py#l69
>
> But that said, in general you're probably right. -Damian

    Thanks for the feedback. You're right, covering all cases is not bad in  
itself. The last point was the weakest, both because I had thought the  
least about it, and because I didn't have an example to back it up.

    The main thing I had in mind when I wrote that point was some test  
cases I had recently reviewed at work. They make some statistical  
computations, and for the tests it *creates* hundreds of test cases (and  
calculates the expected values!) in a loop, then checks the actual values  
are the same as the (calculated) expected. That makes the test code as  
complex as the production code, making it very hard to know if the tests  
are correct, and once something fails, they're pretty hard to debug.

    As I had seen it before, and my impression was that the reason for this  
happening was developers not realising test code is different than  
"production" code (among other things, because it doesn't have to be  
"generic" as in "covering all cases"). In hindsight, after you comment,  
maybe the main/only bad thing about that test was *calculating* the  
expected value.

-- 
Esteban