[tor-dev] gitian replacement proposal

[Nicolas Vigier]

Hi,

On Mon, 13 Jan 2014, Georg Koppen wrote:

> Hi,
> 
> Nicolas Vigier:
> > Hello,
> > 
> > You can find at this URL a proposal to refactor the tor browser bundle
> > build process, using an other tool to replace gitian:
> > https://people.torproject.org/~boklm/automation/tor-automation-proposals.html#build-tool
> > (also added as attached file to this email)
> 
> it seems to be much more than a proposal for refactoring the TBB build
> process as building packages of components is not relevant in the
> latter. So, I won't comment on the details but like to get consensus on
> the big picture first. Taking the main improvements you listed below as
> a starting point seems therefore fine to me:
> 
> > The main improvements in this prototype from the current build process
> > are:
> > 
> > - all components are built separately, and include in their output file
> >   name the commit hash or version, architecture and OS (if architecture
> >   dependant). This allows us to keep previous builds if the
> >   commit/version/architecture/OS didn't change. So we can rebuild a
> >   bundle very quickly when the browser didn't change.
> 
> Yes, we start with #10120 (which I'd like to start working on in the
> next weeks). But avoiding to rebuild other parts of the bundle
> (torbutton etc.) should be easily doable as well (but keep the
> starting/updating/stopping-the-VM-overhead in mind).
> 
> > - the gitian replacment has features to download tarballs and verify them
> >   with sha256sum or gpg signature, so this can replace the fetch-inputs.sh
> >   script.
> 
> Yes, but we already have fetch-inputs.sh. So the advantage of burps
> seems not to be so big here.

The advantage is that it's more simple, and better integrated with the
rest of the tool.

For instance, the python sources tarball is currently defined in the
following files:
 - versions: version and download URL
 - fetch-inputs.sh: gpg signature verification, and creation of a
   symlink Python-$VERSION.tar.bz2 -> python.tar.bz2
 - descriptors/linux/gitian-firefox.yml: python.tar.bz2 is defined in
   the list of files to be copied inside the build VM

In burps the equivalent can be defined in one place, in the file
projects/python/config, with the following lines:

  version: 2.7.5
  input_files:
    - name: python
      filename: 'Python-[% c("version") %].tar.bz2'
      URL: 'http://www.python.org/ftp/python/[% c("version") %]/[% c("filename") %]'
      file_gpg_id: 1
      sig_ext: asc

In this definition we have:
 - the name of the file that should be copied to the build VM (filename)
 - the URL to download the file if it is missing (URL)
 - the 'file_gpg_id' option to indicate that a gpg signature file should
   be downloaded too, and used to verify the file (using keyring python.gpg)
 - no symlink needed. I think it is needed in the gitian build process
   because there is no easy way to access the python version defined in
   the versions file from the gitian descriptor, so a symlink is created
   to avoid updating the filename in the gitian descriptor each time
   the version changes. In burps we can access this filename, so we don't
   need a symlink.

So I think this is more simple. An other advantage is that the files
are going to be downloaded only if they are needed: if we make python
build optional, and we run a build that doesn't need it, it won't be
downloaded.

> 
> > - we can remove the linux/windows/macosx descriptors duplication, and
> >   instead use template directives for the parts that differ between
> >   those builds (it's still possible to use separate files if they differ
> >   completly).
> 
> Yes, that is good. Although I am not sure how much this buys us in a
> full-fledged gitian-like setup. And I guess the gitian people would be
> happy to take patches. :)

The prototype I have made does not support Windows and Mac OS X builds
yet, but I have looked at how it can be implemented.

Instead of having 3 separate descriptor files:
 gitian/descriptors/linux/gitian-tor.yml
 gitian/descriptors/windows/gitian-tor.yml
 gitian/descriptors/mac/gitian-tor.yml

We have only one, but we make changes like this for the parts that
need to differ between Linux / Windows / Mac OS X builds:

  diff --git a/burps.conf b/burps.conf
  index 60d2868be16f..8941b0c7ed94 100644
  --- a/burps.conf
  +++ b/burps.conf
  @@ -36,3 +36,9 @@ targets:
     include_pt:
       var:
          include_pt: 1
  +  win32:
  +    var:
  +       crosscompile_host: i686-w64-mingw32
  +  osx32:
  +    var:
  +       crosscompile_host: i686-apple-darwin11
  diff --git a/projects/tor/build b/projects/tor/build
  index b8cd9f805922..42ac7cf9c67b 100644
  --- a/projects/tor/build
  +++ b/projects/tor/build
  @@ -12,6 +12,9 @@ mkdir "$INSTDIR"
   ./autogen.sh
   [% c('var/touch_directory', { directory => '.' }) %]
   ./configure --disable-asciidoc --with-libevent-dir="$rootdir/libevent" \
  +	    [% IF c('var/crosscompile_host') -%]
  +		--host=[% c('var/crosscompile_host') -%]
  +	    [%- END -%]
   	    --prefix="$INSTDIR"
   make -l[% c('var/max_load') %] -j
   make install

We can then see the different build scripts with:

  $ burps showconf tor build --target dev | grep -A1 configure
  ./configure --disable-asciidoc --with-libevent-dir="$rootdir/libevent" \
                      --prefix="$INSTDIR"
  $ burps showconf tor build --target dev --target win32 | grep -A1 configure
  ./configure --disable-asciidoc --with-libevent-dir="$rootdir/libevent" \
                          --host=i686-w64-mingw32     --prefix="$INSTDIR"

If in the future we want to build for Windows 64 or Mac OS X 64, we can
easily add new targets in burps.conf that set a different value for the
option var/crosscompile_host.

If we want to do the same using gitian, I think the following features
are currently missing:

- possibility to use some templating instructions in the script
  definition. Or some other way to access the info about which OS we
  should build for, from the script.

- possibility to define some options in a common configuration file so
  we can avoid duplicating them in all descriptors.

- possibility to select some OS target on the command line

> 
> > - we can define variables based on selected OS. This allows for instance
> >   to build python 2.7 when building on Ubuntu Lucid, but avoid building
> >   it on other distros which already provide python 2.7.
> 
> Well, building Python already ourselves is actually a feature as we are
> no longer dependent on some Python package. In the very loooooooooong
> run the aim is to build everything ourselves. A better example might
> therefore be building Binutils which we only need for Windows. But that
> boils basically down to #10120 again (we might even be able to be so
> smart to build the platform dependent tools only if the user really
> wants to build for that particular platform: there is no need to build
> Binutils if the user only wants to build Linux bundles).
> 
> > - we can define variables based on "targets" that are set on command
> >   line. For instance in the prototype using "--target enable_pt" instructs
> >   to include the portable transports (only pyptlib in this prototype) in
> >   the bundle.
> 
> The portable transports are supposed to get included into the stable TBB
> rather sooner than later. Thus, that feature is not needed either.

Maybe that won't be needed for portable transports, but we can imagine
having in the future other types of bundles with experimental components
that we only want to have in a separate bundle.

An other use is to enable or disable the build with the randomized
readdir library that was discussed before.

https://lists.torproject.org/pipermail/tor-dev/2013-December/005925.html

> 
> > - we can easily switch from building in a VM to building locally
> 
> True, but I am not sure why that is a feature compared to Gitian as we
> need the VM for creating reproducible builds. Thus, this one does not
> count here as an improvement IMO.

It think this can be useful to be able to easily disable the use of a VM
in some cases:

- we want to do an experimental build with clang instead of GCC. Or a
  different version of GCC / glibc, to understand if a problem is caused
  by Ubuntu GCC / glibc version. An easy way to disable the use of an
  Ubuntu VM can allow to do that.

- in the future, if we're building all the toolchain ourself, we will
  want to check that building on Ubuntu and on other distros produce the
  same result.

> 
> > - build descriptors can depend on the result of another build descriptor.
> >   This remove needs for scripts like mkbundle-*.sh.
> 
> Good idea. And looking at
> https://github.com/bitcoin/bitcoin/tree/master/contrib/gitian-descriptors the
> Bitcoin people might be interested in this as one.
> 
> > And I think those improvements should make it easier to rebuild a new
> > bundle automatically when any of the components of the bundle receives
> > a new commit, and then run tests on this bundle.
> 
> I understand why the first improvement makes rebuilding the bundles
> easier. But why does that hold for the other features as well?
> 
> To get the discussion properly started I think we should ask
> additionally why there has to be a new tool for building TBB. Why not
> improving Gitian? Is it broken beyond repair? Others using Gitian could
> benefit as well and it would save maintenance costs (due to creating yet
> another tool doing similar things *and* maintaining that one etc.). As
> far as I can see none of your improvements is so specific to your tool
> that they can't get included into Gitian. This point is especially worth
> considering as you don't want to get rid of Gitian's functionality
> entirely but only of Gitian for driving the build process if I
> understood that correctly. All those tricky things concerning VM
> creation/handling are kept (and improved :) )

Yes, improving Gitian to have similar features would be an other
solution. However I think it would require important changes in how
Gitian works, and that would be more work, to reimplement the features
that are already available in burps. But it should be possible.

If you're wondering why I did not improve Gitian instead of creating
burps, I can explain that. The main reason is that initialy I did not
intend to make a gitian replacement, I only wanted a tool that would
allow me to automate creation of an rpm package from a software with
its sources in a git repository: git clone/fetch a repository, make a
tarball from a selected commit, create an rpm spec file from a template,
and run rpmbuild to generate the package. I tried to do that in a generic
way so it can be extended to support Debian and other types of packaging.
I'm not running Debian but I wanted to be able to make Debian packages,
so I added an option to be able to build inside a VM/chroot. And added
other options to make it easy to configure and extend.

Later I looked at Gitian more closely, and realized that it was doing
something quite similar to the tool I had been making, but much more
limited. So I started wondering if the tool I made could be used to
build tor browser bundles and quickly made a prototype to check that it
was possible, and see what it would look like and whether it would make
sense to do that.

Now I think burps has all gitian features, with some improvements, and
is much easier to extend with its configuration system and use of
templates. So I think that would be a good change.

> 
> "- creation and start/stop of the Ubuntu build VMs. We can keep the
>   gitian scripts for that, and improve them later."
> 
> So, from my current understanding I tend to think there should be a
> couple of bugs get filed against gitian-builder (and that are good bugs
> you point out, I think!). They should get fixed then and upstreamed.

I think there are two different ways to do it:

- Improve gitian to add all missing features. I think this is difficult
  to do while keeping compatibility with previous versions of gitian, so
  I don't know if upstream will accept the patches. The final result
  will be something similar to what burps based build system is.

- I continue working on a burps based prototype, and make it rebuilt
  automatically by some Jenkins. When this prototype is able to produce
  the same bundles as gitian ones, we use it for the next releases and
  stop updating the gitian based one.

I'm not sure that it is easy to do those changes incrementaly. So my
favorite solution is the 2nd one.

> 
> That said, maybe having the whole packaging in the same tool as well
> changes things, I don't know (it might be worth thinking about the
> additional complexity due to burps being a packaging tool, too: e.g.
> does the lsb_release/release + lsb_release/id combination not matter for
> TBBs). But that is probably a different discussion (or is it not?).

The lsb_release/* is just a way to identify the distribution used for
the build. If you don't want the build to work differently depending on
the distribution, you can ignore that.

The packaging in the same tool is also something interesting I think. In
gitian the build script is defined in the 'script' option inside the
descriptor. In burps the equivalent of that is the 'build' option. But
in the same descriptor file (or files included from the descriptor file),
we can also have an rpm spec file, debian package files. If later we
want to create docker images (www.docker.io), we can easily add docker
files too.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20140117/dd3abc74/attachment-0001.sig>