[tor-dev] bridgdb automation

Thu Dec 19 06:44:53 UTC 2013

On Thu, Dec 19, 2013 at 02:52:03AM +0100, Nicolas Vigier wrote:
> On Tue, 17 Dec 2013, isis agora lovecruft wrote:
> 
> > Nicolas Vigier transcribed 1.4K bytes:

Hi  Nicolas!

Thanks again for following up on this!

> > 
> > Just in case you haven't seen it, Lunar made a wiki page which has quite a bit
> > of info on it, and I filled in some more on BridgeDB. [0]
> 
> Yes, Lunar showed me this page, and we used it when he gave me a summary
> of what each of the projects do.
> 
> > 
> > aabgsn maintained BridgeDB for a year or so, but no longer works on it (though
> > they are more than welcome to do so, if they wish to). sysrqb has been helping
> > me maintain BridgeDB quite a bit (feel free to CC them on BridgeDB topics).
> > 
> > > I am currently looking at the status and list of things to be done
> > > regarding automation on tor project. I have been looking at bridgedb :
> > > https://people.torproject.org/~boklm/automation/tor-automation-review.html#_bridgedb
> > 
> > From that page:
> > 
> > > Continuous Build
> > > BridgeDB is not currently built and tested by Jenkins.
> > > 
> > > However, Isis Lovecruft has a personnal development fork on github that is
> > > built and tested by travis-ci.org:
> > > https://travis-ci.org/isislovecruft/bridgedb/ 
> > >
> > > Packaging
> > > BridgeDB does not have packages. It is currently deployed using a Python virtualenv.
> > >
> > 
> > To my knowledge, BridgeDB is not currently deployed in a virtualenv (sysrqb
> > was the last to redeploy it). I recently refactored the main loop and scripts
> > so that it *can* run in a virtualenv, and it *should* be run in one, because:
> > 
> >   1. We won't need to nag weasel/Sebastian to update/install BridgeDB dependencies.
> >   2. Dependencies will not be installed via sudo.

This sounds advantageous. It's currently running with unmodified PATH,
PYTHONPATH, etc. env vars using the existing scripts to install and
run it. It doesn't install under /usr, so the normal user can
install it.

> 
> I'm not familiar yet with the process to maintain *.tpo services, and
> what part is done by sysadmin team, and what part can be done by
> maintainer of a service, like installing dependencies or other operations
> that require root access. Do you (or someone else reading this) have more
> details about this ?
> 

The general rule I've determined is that if packages are being installed
or upgraded then it's sysadmin, similarly if something is owned by root.
Otherwise the group is responsible for it. I'm probably missing
something important, though.

> > 
> > I've been considering creating packages for BridgeDB on PyPI.
> > 
> >   Pros:
> >     * Even if we manually download the bundle, verify the hash, and then
> >       install it, this seems potentially easier and less error-prone than
> >       checking out a git tag, verifying it, and then building.
> >     * Packaging it now reserves the 'bridgedb' Python namespace for our use.
> > 
> >   Cons:
> >     * I don't want to make people think that this thing is a polished
> >       distribution system for people who wish to run their own BridgeAuths.

1) I don't think we really need to worry about this :)
2) "Please don't deploy this yourself. But, if you do, deploy
carefully, this project is under heavy development"

> > 
> > If proper packaging is helpful for Jenkins, however, I can easily do so.
> 
> An idea could be to have a Debian package for bridgedb, and make Jenkins
> update the packages in a repository automatically when there are new
> commits.
> 

For our purpose I think debian packages are a bit overkill, but I have
nothing against creating them if it will make testing and deployment
easier.

> We could have the following package repositories :
> 
>  - bridgedb-common-wheezy: backports of dependencies required to run
>    bridgedb on wheezy
> 
>  - bridgedb-master-wheezy: packages produced from master (or
>    development) branch on wheezy
>  
>  - bridgedb-stable-wheezy: packages produced from stable branch on
>    wheezy
> 
>  - bridgedb-xxxx-wheezy: packages from xxxx branch, if needed
> 

We also have a develop branch, right now it only resides in our personal
repos, though. This will be useful to test in the future.

> When you push a new commit, Jenkins rebuilds the package and moves it to a
> temporary repository. Then it starts a new container or VM with that
> repository enabled, install the bridgedb package and run some tests. If
> the tests succeed, the package is moved to the repository corresponding
> to the branch.
> 
> When you think that what is in the master branch is ready to be used,
> you merge it to the stable branch, wait for Jenkins to rebuild the
> packages then run apt-get upgrade on the server where bridgedb is
> running. You can also setup a second server that is used as a staging
> environment, with the master branch packages repository.
> 
> It might be possible to do something similar using pip and virtualenv.
> However if it can be done using Debian packages, it will be easier as
> the process to update and test Debian packages can be shared with the
> other projects that are not in python.
> 

In the worst case we can maintain both methods.

> > 
> > > Testing
> > > Some unit tests are implemented in lib/bridgedb/Tests.py and can be run with
> > > the command python setup.py test.
> > 
> > Actually, the tests in lib/bridgedb/Tests.py are old tests. Running them with
> > `python setup.py test` or `make test` will run them via the Python stdlib
> > unittest module (which doesn't play nicely with Twisted's asynchronicity). See
> > #9865, #9872. [1] [2]
> > 
> > There are new tests in lib/bridgedb/test/test_*.py [3] and they can be run with
> > `[sudo] make [re]install && bridgedb test`.
> 
> Ok, so the tests should be run from the installed version. This looks
> nice.
> 

Isis has done a lot of work on this, it's awesome.

> > 
> > I began setting up a system which will run the old lib/bridgedb/Tests.py
> > unittests with Twisted's trial runner (#9873). [4] The old unittests will get
> > run twice, once with removed/deprecated classes and functions which have been
> > taken out of BridgeDB's codebase, and again with new/refactored code; this
> > way, the old unittests function as a (partial) regression test suite.
> > 
> > The way I designed it, the removed/deprecated code (various classes/functions
> > before refactoring) will go into lib/bridgedb/test/deprecated.py, and they are
> > `twisted.python.monkey.MonkeyPatch`ed into place for a run of the old unitests
> > in lib/bridgedb/Tests.py. Then, the old unittests are run a second time with
> > the newly refactored code, so that the difference between the two can be
> > clearly seen, and bugs introduced by new code can (hopefully) be caught
> > immediately.
> > 
> > >
> > > Proposals
> > > Add BridgeDB build and test to Jenkins
> > 
> > Created ticket #10417: BridgeDB should be built and tested on Jenkins 
> > https://trac.torproject.org/projects/tor/ticket/10417
> > 
> > > The main thing to be done that I have seen is running the unit tests
> > > with Jenkins when there are new commits. You can let me know if I missed
> > > something important, or if you have other ideas / needs.
> > 
> > Needs:
> > 
> >   1. We need a lot more unittests, but this is perhaps not a task for
> >      volunteers (or, rather, people who aren't very familiar with BridgeDB's
> >      code).
> >   2. BridgeDB needs *a lot* more documentation. It had almost none when I
> >      started working on it 6 months ago; it has a few bits now. [8]
> > 
> > Questions:
> >   1. Does it help if I use tox? [5] [6]
> >   2. If not, I believe you'll need a shell script which Jenkins can use to
> >      install BridgeDB in a virtualenv. [7] Or do you need some sort of Maven
> >      thing, or both?
> 
> If we use Debian packages as described above, then we don't need
> virtualenv, but we need some Debian packaging.
> 
> >   3. Is there somewhere I should put that documentation on torproject.org
> >      (other than people.tpo/~isis)?
> 
> The https://para.noid.cat/bridgedb/ documentation ?
> 
> I don't know if there is already some place for such documentation.
> Maybe that could be something like http://doc.torproject.org/bridgedb/,
> and we can have it updated by Jenkins automatically when there are new
> commits ?
> 

That sounds like a good idea. Tor has [TOR-DOC] and Stem has [STEM-DOC]
but as far as I know Tor Project doesn't have a single location for
all documentation, which could be nifty and useful for new volunteers.

[TOR-DOC] https://doxygen.torproject.org/
[STEM-DOC] https://stem.torproject.org/

Sadly, I haven't put much thought into this yet, so I will do so and
then try to send you a more useful response.

Thanks for helping us with this.

- Matt