[tor-dev] bridgdb automation

Thu Jan 2 22:08:50 UTC 2014

Nicolas Vigier transcribed 6.8K bytes:
> Hi Matthew and Isis,
> 
> On Sun, 22 Dec 2013, isis agora lovecruft wrote:
> 
> > > > > If proper packaging is helpful for Jenkins, however, I can easily do so.
> > > > 
> > > > An idea could be to have a Debian package for bridgedb, and make Jenkins
> > > > update the packages in a repository automatically when there are new
> > > > commits.
> > > > 
> > > 
> > > For our purpose I think debian packages are a bit overkill, but I have
> > > nothing against creating them if it will make testing and deployment
> > > easier.
> > > 
> > 
> > I am moderately to strongly against using Debian's packaging system for Python
> > things, because it is perpetually outdated, combined with the Python Software
> > Foundation's complete disregard for the standard packaging concept of
> > backporting patches. When something breaks in Python, they fix it in an
> > upcoming release. If you complain that something is broken which is fixed in a
> > newer release than the Python version you're using, the Python devs will tell
> > you to upgrade.
> > 
> > Debian sid's version is currently 2.7.5-5. Which is outdated. 2.7.6 was
> > released two weeks ago. Wheezy is even worse; it's nearly two years
> > outdated. Briefly skimming it, I can point at roughly 30 bugs in the Python
> > release changelog [0] which BridgeDB will likely hit, if we use the wheezy
> > version (which we are using). Several of those bugs due to using ancient,
> > deprecated, OpenSSL API features, and other rather severe SSL bugs, one of
> > which was a recent CVE. (CVE-2013-4238) [1]
> > 
> > [0]: http://hg.python.org/cpython/raw-file/99d03261c1ba/Misc/NEWS
> > [1]: https://security-tracker.debian.org/tracker/CVE-2013-4238
> > 
> > What is more important, and what I would *really* prefer not to fight, is the
> > inevitable slew of horrific glitches and hiccups which will occur from using
> > Debian's extremely outdated Python modules. Here is a list of Python packages
> > in Debian, compared to the version in Ubuntu, along with excuses for why the
> > haven't been updated. [2] Twisted in wheezy is also about two years
> > outdated. Jessie/sid aren't up-to-date either. This is madness.
> > 
> > [2]: http://qa.debian.org/developer.php?login=python-modules-team@lists.alioth.debian.org
> > 
> > I don't mean to start a holy war, and I love Debian… but their Python team is
> > failing. I think the only reason Debian managed to get a somewhat safe version
> > of Pip (v1.4.1) into jessie and sid is that one of the main organisers in
> > Debian Security convinced me to file CVEs for vulnerabilities in pip-1.3.1,
> > even though dstufft had already fixed the issues in 1.4.1. If I understood the
> > politics correctly, they needed me to file those CVEs so that the security
> > team could update the packages themselves ― without waiting for the lead of
> > the Debian Python group to give the okay (the latter being the person who
> > seems to be causing most of these problems; incidentally, that person works
> > for Canonical).
> 
> Ok, I didn't know that, thanks for the details. So let's avoid Debian
> packages for BridgeDB for now.
> 
> > 
> > If, somehow, this process involves staging a violent revolutionary upheaval of
> > the current lead of the Debian Python team, and whichever other jerks are
> > preventing Debian from having decent Python packaging… count me in! Otherwise
> > I'd really rather steer clear of them.
> > 
> > > > We could have the following package repositories :
> > > > 
> > > >  - bridgedb-common-wheezy: backports of dependencies required to run
> > > >    bridgedb on wheezy
> > > > 
> > > >  - bridgedb-master-wheezy: packages produced from master (or
> > > >    development) branch on wheezy
> > > >  
> > > >  - bridgedb-stable-wheezy: packages produced from stable branch on
> > > >    wheezy
> > > > 
> > > >  - bridgedb-xxxx-wheezy: packages from xxxx branch, if needed
> > > > 
> > > 
> > > We also have a develop branch, right now it only resides in our personal
> > > repos, though. This will be useful to test in the future.
> > > 
> > 
> > Develop is probably the branch to test. I don't ever make commits straight
> > onto master branches in any repo.
> > 
> > > > When you push a new commit, Jenkins rebuilds the package and moves it to a
> > > > temporary repository. Then it starts a new container or VM with that
> > > > repository enabled, install the bridgedb package and run some tests. If
> > > > the tests succeed, the package is moved to the repository corresponding
> > > > to the branch.
> > 
> > Does "repository" here mean "Debian repository"? Or "directory of tarballs and
> > sha256sums"?
> 
> I was thinking about a Debian repository when writting this, but this
> can be done as a directory of tarballs and sha256sums if that's better.
> Is it something that pip can use ?

Yep! Pip will use that just fine.

> But maybe that is not very useful as it seems pip can install from a git
> URL directly. 

Or that. Whichever is easier.

> > > > When you think that what is in the master branch is ready to be used,
> > > > you merge it to the stable branch, wait for Jenkins to rebuild the
> > > > packages
> > 
> > This is done! With the slight modification that my master = your stable, and
> > my develop = your master, my fix/* and feature/* are all branches for specific
> > tickets, and my testing/* branches are all staging branches (from fix/* and
> > feature/*) which will get merged into my develop.
> > 
> > The main idea being that someone pulling from master should always get the
> > latest stable release.
> 
> Ok. So to summarize the things that would be useful to do, we have:
> 
> - run tests from Jenkins, with different python versions, when there are
>   new commits in the master branch. We do that by creating a virtualenv,
>   installing BridgeDB in the virtualenv and runnig "bridgedb test".
>   (#10417)
> 
> - do the same for the develop branch in user repositories
> 
> - install a staging instance on ponticum
> 
> - make jenkins build the docs from the master branch, and upload them
>   somewhere like docs.tpo/bridgedb

Sounds good.

-- 
 ♥Ⓐ isis agora lovecruft
_________________________________________________________
GPG: 4096R/A3ADB67A2CDB8B35
Current Keys: https://blog.patternsinthevoid.net/isis.txt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1154 bytes
Desc: Digital signature
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20140102/fc98190a/attachment.sig>