[tor-dev] gitian replacement proposal

[Georg Koppen]

Hi,

Nicolas Vigier:
> Hello,
> 
> You can find at this URL a proposal to refactor the tor browser bundle
> build process, using an other tool to replace gitian:
> https://people.torproject.org/~boklm/automation/tor-automation-proposals.html#build-tool
> (also added as attached file to this email)

it seems to be much more than a proposal for refactoring the TBB build
process as building packages of components is not relevant in the
latter. So, I won't comment on the details but like to get consensus on
the big picture first. Taking the main improvements you listed below as
a starting point seems therefore fine to me:

> The main improvements in this prototype from the current build process
> are:
> 
> - all components are built separately, and include in their output file
>   name the commit hash or version, architecture and OS (if architecture
>   dependant). This allows us to keep previous builds if the
>   commit/version/architecture/OS didn't change. So we can rebuild a
>   bundle very quickly when the browser didn't change.

Yes, we start with #10120 (which I'd like to start working on in the
next weeks). But avoiding to rebuild other parts of the bundle
(torbutton etc.) should be easily doable as well (but keep the
starting/updating/stopping-the-VM-overhead in mind).

> - the gitian replacment has features to download tarballs and verify them
>   with sha256sum or gpg signature, so this can replace the fetch-inputs.sh
>   script.

Yes, but we already have fetch-inputs.sh. So the advantage of burps
seems not to be so big here.

> - we can remove the linux/windows/macosx descriptors duplication, and
>   instead use template directives for the parts that differ between
>   those builds (it's still possible to use separate files if they differ
>   completly).

Yes, that is good. Although I am not sure how much this buys us in a
full-fledged gitian-like setup. And I guess the gitian people would be
happy to take patches. :)

> - we can define variables based on selected OS. This allows for instance
>   to build python 2.7 when building on Ubuntu Lucid, but avoid building
>   it on other distros which already provide python 2.7.

Well, building Python already ourselves is actually a feature as we are
no longer dependent on some Python package. In the very loooooooooong
run the aim is to build everything ourselves. A better example might
therefore be building Binutils which we only need for Windows. But that
boils basically down to #10120 again (we might even be able to be so
smart to build the platform dependent tools only if the user really
wants to build for that particular platform: there is no need to build
Binutils if the user only wants to build Linux bundles).

> - we can define variables based on "targets" that are set on command
>   line. For instance in the prototype using "--target enable_pt" instructs
>   to include the portable transports (only pyptlib in this prototype) in
>   the bundle.

The portable transports are supposed to get included into the stable TBB
rather sooner than later. Thus, that feature is not needed either.

> - we can easily switch from building in a VM to building locally

True, but I am not sure why that is a feature compared to Gitian as we
need the VM for creating reproducible builds. Thus, this one does not
count here as an improvement IMO.

> - build descriptors can depend on the result of another build descriptor.
>   This remove needs for scripts like mkbundle-*.sh.

Good idea. And looking at
https://github.com/bitcoin/bitcoin/tree/master/contrib/gitian-descriptors the
Bitcoin people might be interested in this as one.

> And I think those improvements should make it easier to rebuild a new
> bundle automatically when any of the components of the bundle receives
> a new commit, and then run tests on this bundle.

I understand why the first improvement makes rebuilding the bundles
easier. But why does that hold for the other features as well?

To get the discussion properly started I think we should ask
additionally why there has to be a new tool for building TBB. Why not
improving Gitian? Is it broken beyond repair? Others using Gitian could
benefit as well and it would save maintenance costs (due to creating yet
another tool doing similar things *and* maintaining that one etc.). As
far as I can see none of your improvements is so specific to your tool
that they can't get included into Gitian. This point is especially worth
considering as you don't want to get rid of Gitian's functionality
entirely but only of Gitian for driving the build process if I
understood that correctly. All those tricky things concerning VM
creation/handling are kept (and improved :) )

"- creation and start/stop of the Ubuntu build VMs. We can keep the
  gitian scripts for that, and improve them later."

So, from my current understanding I tend to think there should be a
couple of bugs get filed against gitian-builder (and that are good bugs
you point out, I think!). They should get fixed then and upstreamed.

That said, maybe having the whole packaging in the same tool as well
changes things, I don't know (it might be worth thinking about the
additional complexity due to burps being a packaging tool, too: e.g.
does the lsb_release/release + lsb_release/id combination not matter for
TBBs). But that is probably a different discussion (or is it not?).

Georg




-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 862 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20140113/e0a09b3b/attachment.sig>