[tor-dev] bridgdb automation

Tue Dec 17 21:58:34 UTC 2013

Nicolas Vigier transcribed 1.4K bytes:
> Hi,

Hey Nicolas,

Thanks for all the work you're doing, and my apologies that I hadn't responded
to your tor-dev@ call yet.

Just in case you haven't seen it, Lunar made a wiki page which has quite a bit
of info on it, and I filled in some more on BridgeDB. [0]

aabgsn maintained BridgeDB for a year or so, but no longer works on it (though
they are more than welcome to do so, if they wish to). sysrqb has been helping
me maintain BridgeDB quite a bit (feel free to CC them on BridgeDB topics).

> I am currently looking at the status and list of things to be done
> regarding automation on tor project. I have been looking at bridgedb :
> https://people.torproject.org/~boklm/automation/tor-automation-review.html#_bridgedb

From that page:

> Continuous Build
> BridgeDB is not currently built and tested by Jenkins.
> 
> However, Isis Lovecruft has a personnal development fork on github that is
> built and tested by travis-ci.org:
> https://travis-ci.org/isislovecruft/bridgedb/ 
>
> Packaging
> BridgeDB does not have packages. It is currently deployed using a Python virtualenv.
>

To my knowledge, BridgeDB is not currently deployed in a virtualenv (sysrqb
was the last to redeploy it). I recently refactored the main loop and scripts
so that it *can* run in a virtualenv, and it *should* be run in one, because:

  1. We won't need to nag weasel/Sebastian to update/install BridgeDB dependencies.
  2. Dependencies will not be installed via sudo.

I've been considering creating packages for BridgeDB on PyPI.

  Pros:
    * Even if we manually download the bundle, verify the hash, and then
      install it, this seems potentially easier and less error-prone than
      checking out a git tag, verifying it, and then building.
    * Packaging it now reserves the 'bridgedb' Python namespace for our use.

  Cons:
    * I don't want to make people think that this thing is a polished
      distribution system for people who wish to run their own BridgeAuths.

If proper packaging is helpful for Jenkins, however, I can easily do so.

> Testing
> Some unit tests are implemented in lib/bridgedb/Tests.py and can be run with
> the command python setup.py test.

Actually, the tests in lib/bridgedb/Tests.py are old tests. Running them with
`python setup.py test` or `make test` will run them via the Python stdlib
unittest module (which doesn't play nicely with Twisted's asynchronicity). See
#9865, #9872. [1] [2]

There are new tests in lib/bridgedb/test/test_*.py [3] and they can be run with
`[sudo] make [re]install && bridgedb test`.

I began setting up a system which will run the old lib/bridgedb/Tests.py
unittests with Twisted's trial runner (#9873). [4] The old unittests will get
run twice, once with removed/deprecated classes and functions which have been
taken out of BridgeDB's codebase, and again with new/refactored code; this
way, the old unittests function as a (partial) regression test suite.

The way I designed it, the removed/deprecated code (various classes/functions
before refactoring) will go into lib/bridgedb/test/deprecated.py, and they are
`twisted.python.monkey.MonkeyPatch`ed into place for a run of the old unitests
in lib/bridgedb/Tests.py. Then, the old unittests are run a second time with
the newly refactored code, so that the difference between the two can be
clearly seen, and bugs introduced by new code can (hopefully) be caught
immediately.

>
> Proposals
> Add BridgeDB build and test to Jenkins

Created ticket #10417: BridgeDB should be built and tested on Jenkins 
https://trac.torproject.org/projects/tor/ticket/10417

> The main thing to be done that I have seen is running the unit tests
> with Jenkins when there are new commits. You can let me know if I missed
> something important, or if you have other ideas / needs.

Needs:

  1. We need a lot more unittests, but this is perhaps not a task for
     volunteers (or, rather, people who aren't very familiar with BridgeDB's
     code).
  2. BridgeDB needs *a lot* more documentation. It had almost none when I
     started working on it 6 months ago; it has a few bits now. [8]

Questions:
  1. Does it help if I use tox? [5] [6]
  2. If not, I believe you'll need a shell script which Jenkins can use to
     install BridgeDB in a virtualenv. [7] Or do you need some sort of Maven
     thing, or both?
  3. Is there somewhere I should put that documentation on torproject.org
     (other than people.tpo/~isis)?

[0]: https://trac.torproject.org/projects/tor/wiki/AutomationInventory
[1]: https://trac.torproject.org/projects/tor/ticket/9865
[2]: https://trac.torproject.org/projects/tor/ticket/9872
[3]: https://gitweb.torproject.org/user/isis/bridgedb.git/tree/refs/heads/develop:/lib/bridgedb/test
[4]: https://trac.torproject.org/projects/tor/ticket/9873
[5]: http://tox.readthedocs.org/en/latest/
[6]: http://alexgaynor.net/2010/dec/17/getting-most-out-tox/
[7]: http://www.alexconrad.org/2011/10/jenkins-and-python.html
[8]: https://para.noid.cat/bridgedb/

Thanks,
-- 
 ♥Ⓐ isis agora lovecruft
_________________________________________________________
GPG: 4096R/A3ADB67A2CDB8B35
Current Keys: https://blog.patternsinthevoid.net/isis.txt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1154 bytes
Desc: Digital signature
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20131217/3cd047cb/attachment-0001.sig>