[tor-dev] bridgdb automation

Thu Dec 19 01:52:03 UTC 2013

On Tue, 17 Dec 2013, isis agora lovecruft wrote:

> Nicolas Vigier transcribed 1.4K bytes:
> > Hi,
> 
> Hey Nicolas,
> 
> Thanks for all the work you're doing, and my apologies that I hadn't responded
> to your tor-dev@ call yet.

Hi Isis, thanks for your answer.

> 
> Just in case you haven't seen it, Lunar made a wiki page which has quite a bit
> of info on it, and I filled in some more on BridgeDB. [0]

Yes, Lunar showed me this page, and we used it when he gave me a summary
of what each of the projects do.

> 
> aabgsn maintained BridgeDB for a year or so, but no longer works on it (though
> they are more than welcome to do so, if they wish to). sysrqb has been helping
> me maintain BridgeDB quite a bit (feel free to CC them on BridgeDB topics).
> 
> > I am currently looking at the status and list of things to be done
> > regarding automation on tor project. I have been looking at bridgedb :
> > https://people.torproject.org/~boklm/automation/tor-automation-review.html#_bridgedb
> 
> From that page:
> 
> > Continuous Build
> > BridgeDB is not currently built and tested by Jenkins.
> > 
> > However, Isis Lovecruft has a personnal development fork on github that is
> > built and tested by travis-ci.org:
> > https://travis-ci.org/isislovecruft/bridgedb/ 
> >
> > Packaging
> > BridgeDB does not have packages. It is currently deployed using a Python virtualenv.
> >
> 
> To my knowledge, BridgeDB is not currently deployed in a virtualenv (sysrqb
> was the last to redeploy it). I recently refactored the main loop and scripts
> so that it *can* run in a virtualenv, and it *should* be run in one, because:
> 
>   1. We won't need to nag weasel/Sebastian to update/install BridgeDB dependencies.
>   2. Dependencies will not be installed via sudo.

I'm not familiar yet with the process to maintain *.tpo services, and
what part is done by sysadmin team, and what part can be done by
maintainer of a service, like installing dependencies or other operations
that require root access. Do you (or someone else reading this) have more
details about this ?

> 
> I've been considering creating packages for BridgeDB on PyPI.
> 
>   Pros:
>     * Even if we manually download the bundle, verify the hash, and then
>       install it, this seems potentially easier and less error-prone than
>       checking out a git tag, verifying it, and then building.
>     * Packaging it now reserves the 'bridgedb' Python namespace for our use.
> 
>   Cons:
>     * I don't want to make people think that this thing is a polished
>       distribution system for people who wish to run their own BridgeAuths.
> 
> If proper packaging is helpful for Jenkins, however, I can easily do so.

An idea could be to have a Debian package for bridgedb, and make Jenkins
update the packages in a repository automatically when there are new
commits.

We could have the following package repositories :

 - bridgedb-common-wheezy: backports of dependencies required to run
   bridgedb on wheezy

 - bridgedb-master-wheezy: packages produced from master (or
   development) branch on wheezy

 - bridgedb-stable-wheezy: packages produced from stable branch on
   wheezy

 - bridgedb-xxxx-wheezy: packages from xxxx branch, if needed

When you push a new commit, Jenkins rebuilds the package and moves it to a
temporary repository. Then it starts a new container or VM with that
repository enabled, install the bridgedb package and run some tests. If
the tests succeed, the package is moved to the repository corresponding
to the branch.

When you think that what is in the master branch is ready to be used,
you merge it to the stable branch, wait for Jenkins to rebuild the
packages then run apt-get upgrade on the server where bridgedb is
running. You can also setup a second server that is used as a staging
environment, with the master branch packages repository.

It might be possible to do something similar using pip and virtualenv.
However if it can be done using Debian packages, it will be easier as
the process to update and test Debian packages can be shared with the
other projects that are not in python.

> 
> > Testing
> > Some unit tests are implemented in lib/bridgedb/Tests.py and can be run with
> > the command python setup.py test.
> 
> Actually, the tests in lib/bridgedb/Tests.py are old tests. Running them with
> `python setup.py test` or `make test` will run them via the Python stdlib
> unittest module (which doesn't play nicely with Twisted's asynchronicity). See
> #9865, #9872. [1] [2]
> 
> There are new tests in lib/bridgedb/test/test_*.py [3] and they can be run with
> `[sudo] make [re]install && bridgedb test`.

Ok, so the tests should be run from the installed version. This looks
nice.

> 
> I began setting up a system which will run the old lib/bridgedb/Tests.py
> unittests with Twisted's trial runner (#9873). [4] The old unittests will get
> run twice, once with removed/deprecated classes and functions which have been
> taken out of BridgeDB's codebase, and again with new/refactored code; this
> way, the old unittests function as a (partial) regression test suite.
> 
> The way I designed it, the removed/deprecated code (various classes/functions
> before refactoring) will go into lib/bridgedb/test/deprecated.py, and they are
> `twisted.python.monkey.MonkeyPatch`ed into place for a run of the old unitests
> in lib/bridgedb/Tests.py. Then, the old unittests are run a second time with
> the newly refactored code, so that the difference between the two can be
> clearly seen, and bugs introduced by new code can (hopefully) be caught
> immediately.
> 
> >
> > Proposals
> > Add BridgeDB build and test to Jenkins
> 
> Created ticket #10417: BridgeDB should be built and tested on Jenkins 
> https://trac.torproject.org/projects/tor/ticket/10417
> 
> > The main thing to be done that I have seen is running the unit tests
> > with Jenkins when there are new commits. You can let me know if I missed
> > something important, or if you have other ideas / needs.
> 
> Needs:
> 
>   1. We need a lot more unittests, but this is perhaps not a task for
>      volunteers (or, rather, people who aren't very familiar with BridgeDB's
>      code).
>   2. BridgeDB needs *a lot* more documentation. It had almost none when I
>      started working on it 6 months ago; it has a few bits now. [8]
> 
> Questions:
>   1. Does it help if I use tox? [5] [6]
>   2. If not, I believe you'll need a shell script which Jenkins can use to
>      install BridgeDB in a virtualenv. [7] Or do you need some sort of Maven
>      thing, or both?

If we use Debian packages as described above, then we don't need
virtualenv, but we need some Debian packaging.

>   3. Is there somewhere I should put that documentation on torproject.org
>      (other than people.tpo/~isis)?

The https://para.noid.cat/bridgedb/ documentation ?

I don't know if there is already some place for such documentation.
Maybe that could be something like http://doc.torproject.org/bridgedb/,
and we can have it updated by Jenkins automatically when there are new
commits ?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20131219/42c84f6d/attachment.sig>