[tor-dev] bridgdb automation

[Nicolas Vigier]

Hi Matthew and Isis,

On Sun, 22 Dec 2013, isis agora lovecruft wrote:

> > > > If proper packaging is helpful for Jenkins, however, I can easily do so.
> > > 
> > > An idea could be to have a Debian package for bridgedb, and make Jenkins
> > > update the packages in a repository automatically when there are new
> > > commits.
> > > 
> > 
> > For our purpose I think debian packages are a bit overkill, but I have
> > nothing against creating them if it will make testing and deployment
> > easier.
> > 
> 
> I am moderately to strongly against using Debian's packaging system for Python
> things, because it is perpetually outdated, combined with the Python Software
> Foundation's complete disregard for the standard packaging concept of
> backporting patches. When something breaks in Python, they fix it in an
> upcoming release. If you complain that something is broken which is fixed in a
> newer release than the Python version you're using, the Python devs will tell
> you to upgrade.
> 
> Debian sid's version is currently 2.7.5-5. Which is outdated. 2.7.6 was
> released two weeks ago. Wheezy is even worse; it's nearly two years
> outdated. Briefly skimming it, I can point at roughly 30 bugs in the Python
> release changelog [0] which BridgeDB will likely hit, if we use the wheezy
> version (which we are using). Several of those bugs due to using ancient,
> deprecated, OpenSSL API features, and other rather severe SSL bugs, one of
> which was a recent CVE. (CVE-2013-4238) [1]
> 
> [0]: http://hg.python.org/cpython/raw-file/99d03261c1ba/Misc/NEWS
> [1]: https://security-tracker.debian.org/tracker/CVE-2013-4238
> 
> What is more important, and what I would *really* prefer not to fight, is the
> inevitable slew of horrific glitches and hiccups which will occur from using
> Debian's extremely outdated Python modules. Here is a list of Python packages
> in Debian, compared to the version in Ubuntu, along with excuses for why the
> haven't been updated. [2] Twisted in wheezy is also about two years
> outdated. Jessie/sid aren't up-to-date either. This is madness.
> 
> [2]: http://qa.debian.org/developer.php?login=python-modules-team@lists.alioth.debian.org
> 
> I don't mean to start a holy war, and I love Debian… but their Python team is
> failing. I think the only reason Debian managed to get a somewhat safe version
> of Pip (v1.4.1) into jessie and sid is that one of the main organisers in
> Debian Security convinced me to file CVEs for vulnerabilities in pip-1.3.1,
> even though dstufft had already fixed the issues in 1.4.1. If I understood the
> politics correctly, they needed me to file those CVEs so that the security
> team could update the packages themselves ― without waiting for the lead of
> the Debian Python group to give the okay (the latter being the person who
> seems to be causing most of these problems; incidentally, that person works
> for Canonical).

Ok, I didn't know that, thanks for the details. So let's avoid Debian
packages for BridgeDB for now.

> 
> If, somehow, this process involves staging a violent revolutionary upheaval of
> the current lead of the Debian Python team, and whichever other jerks are
> preventing Debian from having decent Python packaging… count me in! Otherwise
> I'd really rather steer clear of them.
> 
> > > We could have the following package repositories :
> > > 
> > >  - bridgedb-common-wheezy: backports of dependencies required to run
> > >    bridgedb on wheezy
> > > 
> > >  - bridgedb-master-wheezy: packages produced from master (or
> > >    development) branch on wheezy
> > >  
> > >  - bridgedb-stable-wheezy: packages produced from stable branch on
> > >    wheezy
> > > 
> > >  - bridgedb-xxxx-wheezy: packages from xxxx branch, if needed
> > > 
> > 
> > We also have a develop branch, right now it only resides in our personal
> > repos, though. This will be useful to test in the future.
> > 
> 
> Develop is probably the branch to test. I don't ever make commits straight
> onto master branches in any repo.
> 
> > > When you push a new commit, Jenkins rebuilds the package and moves it to a
> > > temporary repository. Then it starts a new container or VM with that
> > > repository enabled, install the bridgedb package and run some tests. If
> > > the tests succeed, the package is moved to the repository corresponding
> > > to the branch.
> 
> Does "repository" here mean "Debian repository"? Or "directory of tarballs and
> sha256sums"?

I was thinking about a Debian repository when writting this, but this
can be done as a directory of tarballs and sha256sums if that's better.
Is it something that pip can use ?

But maybe that is not very useful as it seems pip can install from a git
URL directly. 

> 
> > > When you think that what is in the master branch is ready to be used,
> > > you merge it to the stable branch, wait for Jenkins to rebuild the
> > > packages
> 
> This is done! With the slight modification that my master = your stable, and
> my develop = your master, my fix/* and feature/* are all branches for specific
> tickets, and my testing/* branches are all staging branches (from fix/* and
> feature/*) which will get merged into my develop.
> 
> The main idea being that someone pulling from master should always get the
> latest stable release.

Ok. So to summarize the things that would be useful to do, we have:

- run tests from Jenkins, with different python versions, when there are
  new commits in the master branch. We do that by creating a virtualenv,
  installing BridgeDB in the virtualenv and runnig "bridgedb test".
  (#10417)

- do the same for the develop branch in user repositories

- install a staging instance on ponticum

- make jenkins build the docs from the master branch, and upload them
  somewhere like docs.tpo/bridgedb

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20131224/2a9217ae/attachment-0001.sig>