[tor-dev] Pluggable Transport TBB Build

[David Fifield]

On Sat, Oct 05, 2013 at 11:08:40AM -0400, Kevin P Dyer wrote:
> Sorry I missed the most recent Pluggable Transport bi-weekly meeting.
> 
> In regards to the PTTBB build, I plan to make progress towards a
> streamlined build process.
> 
> Platforms I want to get the PTTBB build working on:
> 1) CentOS 5.9 (32-bit)
> 2) CentOS 5.9 (64-bit)
> 3) Windows 7
> 4) OSX 10.8
> 
> For each build platform I'm going to:
> 1) Create a VirtualBox image.
> 2) Construct a script that takes the latest TBB release and VirtualBox
> image as input, and outputs a PTTBB zip file. The output PTTBB zip
> file will include all of the "deployed" pluggable transports.
> 
> The build process must be deterministic and reproducible.

Thank you for taking an interest in this. Two things:
 1. There already exist build scripts and VM instructions, which we use
    to build the PT TBB. If you want to build a bundle including FTE,
    your best bet is to start by modifying those scripts.
 2. The Tor Project already has a very nice reproducible build system,
    not yet used for the PT TBB. I want to start using it for the PT
    bundles in the nearish future.

For the scripts we use to build the PT TBB now, please see
https://gitweb.torproject.org/pluggable-transports/bundle.git
https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/Makefile
https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bundle-gnulinux.txt
https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bundle-macosx.txt
https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bundle-windows.txt
They work much the way you describe: you boot a VM, and run a "make"
command. The makefile unzips the vanilla bundle, builds the pluggable
transports and copies them into the bundle, then zips it up again.

At the time we started making PT TBBs, Tor's reproducible build system
was not finished. I think the new system has great advantages for PT TBB
builds, so I want to start building them that way. That is the subject
of this ticket:
https://trac.torproject.org/projects/tor/ticket/9444

About the reproducible build system, please see this blog post and its
linked documentation:
https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details
You make a good point about the need for reproducibility. This existing
system took six months of work working around nontrivial problems (see
the blog post), and it's super nice. It's not something you want to
reinvent by yourself.

> I was hoping to do this on Amazon Web Services, to avoid the licensing
> issues. However, AWS does not support OSX.

A huge advantage of the reproducible build system is that it requires
neither a Windows license nor an OS X install. Everything is
cross-compiled from Ubuntu. From a practical perspective, I care about
this feature even more than reproducibility. The need to boot up an
actual physical Mac is one reason why PT builds have lagged behind
(#9391).

> I anticipate this will be a non-trivial effort to get working
> correctly. So, it would be great if you could help me adjust my plan
> to minimize headache!

As you see, it's a bit muddled because we are in a transition between
two build systems. What I recommend is first doing a proof-of-concept
using the bundle-gnulinux.txt instructions (because they are the
easiest). Building the base VM image takes about an hour, and then
running "make" to build a bundle takes under five minutes. (Technically
you don't even need to use a VM; we do so mainly for filesystem
hygiene reasons.)

You should create a new ticket "Add FTE to pluggable transports bundle"
and Cc me. There you can attach patches or link to the repo you are
working in. I understand there are other issues preventing FTE from
being included right away, but I would have no problem with there being
a branch so that it is easy to build experimental bundles including FTE.

As for the transition to the reproducible build system (#9444), your
help would be appreciated with that. As I understand it, what we need to
do is create one or more new Gitian "descriptors" for the pluggable
transports, then modify gitian-bundle.yml to copy them into the bundle.
https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/README.build
https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/descriptors/linux/gitian-tor.yml
https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/descriptors/linux/gitian-firefox.yml
https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/descriptors/linux/gitian-bundle.yml
So far, the most progress I have made on #9444 is to do a reproducible
build of the vanilla bundle.

David Fifield