[tor-dev] lcov test coverage reports

[Nick Mathewson]

On Sun, Jun 15, 2014 at 8:00 AM, David Murray <spam at kdmurray.id.au> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> Would there be interest in using lcov [0] for test coverage analysis?
>
> Obviously, as lcov is an external pacakge, the existing scripts that
> only use built in gcov commands must remain.
>
> However, if I were to create a script that generates a pretty, HTML
> coverage report using lcov & genhtml, would this be something others
> would use? And how would you like to see this implemented? As a custom
> make target (e.g., make htmlcoverage + autotools black magic to silently
> disable/hide the target if we can't find lcov), or as a standalone
> script (i.e., ./scripts/test/run_lcov.sh).

So, I think that if I'm reading the documentation right, all that
should be necessary would be an additional make target.  (I'm assuming
that the make target wouldn't call "configure" -- we shouldn't define
our make targets that do that, IMO.)

(If I'm reading this properly, lcov uses the same command-line
arguments to gcc that gcov does, so it could also depend on the
--enable-coverage configure flag.)

I don't think it's necessary to have the target disabled when there is
no lcov, though it would be nice if it gave a useful error message
instead of "lcov: command not found".


Additionally, one thing that I'd really love to see -- though I don't
at all know whether lcov can do this as it stands -- is a semantic
diff between two coverage outputs.  When writing new unit tests at
random, or when checking functions that you know need coverage, it's
useful to ask "Which lines are not covered?"  But for validating new
code and checking patches, the question I want to ask is "Which *new*
or *changed* lines are not covered?"

We kinda have a script for this now, but learning to read its outputs
is not as easy as I'd like.

cheers,
-- 
Nick