[tor-dev] Pluggable Transport TBB Build

Wed Oct 9 16:02:20 UTC 2013

Hi David!

Thanks for the detailed response. My thoughts are inline.

On Sun, Oct 6, 2013 at 3:30 PM, David Fifield <david at bamsoftware.com> wrote:
> On Sat, Oct 05, 2013 at 11:08:40AM -0400, Kevin P Dyer wrote:
>> Sorry I missed the most recent Pluggable Transport bi-weekly meeting.
>>
>> In regards to the PTTBB build, I plan to make progress towards a
>> streamlined build process.
>>
>> Platforms I want to get the PTTBB build working on:
>> 1) CentOS 5.9 (32-bit)
>> 2) CentOS 5.9 (64-bit)
>> 3) Windows 7
>> 4) OSX 10.8
>>
>> For each build platform I'm going to:
>> 1) Create a VirtualBox image.
>> 2) Construct a script that takes the latest TBB release and VirtualBox
>> image as input, and outputs a PTTBB zip file. The output PTTBB zip
>> file will include all of the "deployed" pluggable transports.
>>
>> The build process must be deterministic and reproducible.
>
> Thank you for taking an interest in this. Two things:
>  1. There already exist build scripts and VM instructions, which we use
>     to build the PT TBB. If you want to build a bundle including FTE,
>     your best bet is to start by modifying those scripts.
>  2. The Tor Project already has a very nice reproducible build system,
>     not yet used for the PT TBB. I want to start using it for the PT
>     bundles in the nearish future.
>
> For the scripts we use to build the PT TBB now, please see
> https://gitweb.torproject.org/pluggable-transports/bundle.git
> https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/Makefile
> https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bundle-gnulinux.txt
> https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bundle-macosx.txt
> https://gitweb.torproject.org/pluggable-transports/bundle.git/blob/HEAD:/bundle-windows.txt
> They work much the way you describe: you boot a VM, and run a "make"
> command. The makefile unzips the vanilla bundle, builds the pluggable
> transports and copies them into the bundle, then zips it up again.

This is a good start and goes a long way towards automating the build
process. However, I'm looking for "Press a button, make a sandwich,
have all builds (for all platforms) ready when I come back."-type of
automation. It appears that the current procedure requires booting up
four systems (virtual or physical) manually, then manually running the
build script in each instance.

I've had success fully-automating the build using vagrant for the
linux builds. (Execute "vagrant up," wait a few minutes, then we have
i386/amd64 PTTBB .tar.gz files.) I only encountered two issues [1,2].
OSX is next. How may I share this work on
https://gitweb.torproject.org/?

> At the time we started making PT TBBs, Tor's reproducible build system
> was not finished. I think the new system has great advantages for PT TBB
> builds, so I want to start building them that way. That is the subject
> of this ticket:
> https://trac.torproject.org/projects/tor/ticket/9444
>
> About the reproducible build system, please see this blog post and its
> linked documentation:
> https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details
> You make a good point about the need for reproducibility. This existing
> system took six months of work working around nontrivial problems (see
> the blog post), and it's super nice. It's not something you want to
> reinvent by yourself.

I agree. We should definitely work towards using that system long term.

>> I was hoping to do this on Amazon Web Services, to avoid the licensing
>> issues. However, AWS does not support OSX.
>
> A huge advantage of the reproducible build system is that it requires
> neither a Windows license nor an OS X install. Everything is
> cross-compiled from Ubuntu. From a practical perspective, I care about
> this feature even more than reproducibility. The need to boot up an
> actual physical Mac is one reason why PT builds have lagged behind
> (#9391).

Cross-compilation would certainly be a nice feature to have. However,
for testing the product of a cross-compiled build, I'll need to have a
development environment with each of the target platforms. I'm
unwilling to have a bunch of physical machines for testing. Hence, I'm
eager for full virtualization, which also helps with full-automation
of the build.

I've looked a bit more into the OSX EULA. Even though running OSX as a
guest on Windows/Linux seems prevalent, it is against the EULA [4].
However, the EULA, since 10.7, states (c.f., sec. 2.B.iii) that you
can spin up virtual instances on Mac hardware. So, I guess it will be
up to each of us to figure out our ideal configuration, but the only
legal way to do full virtualization requires Mac hardware. Lame.

>> I anticipate this will be a non-trivial effort to get working
>> correctly. So, it would be great if you could help me adjust my plan
>> to minimize headache!
>
> As you see, it's a bit muddled because we are in a transition between
> two build systems. What I recommend is first doing a proof-of-concept
> using the bundle-gnulinux.txt instructions (because they are the
> easiest). Building the base VM image takes about an hour, and then
> running "make" to build a bundle takes under five minutes. (Technically
> you don't even need to use a VM; we do so mainly for filesystem
> hygiene reasons.)
>
> You should create a new ticket "Add FTE to pluggable transports bundle"
> and Cc me. There you can attach patches or link to the repo you are
> working in. I understand there are other issues preventing FTE from
> being included right away, but I would have no problem with there being
> a branch so that it is easy to build experimental bundles including FTE.

Done [3]!

> As for the transition to the reproducible build system (#9444), your
> help would be appreciated with that. As I understand it, what we need to
> do is create one or more new Gitian "descriptors" for the pluggable
> transports, then modify gitian-bundle.yml to copy them into the bundle.
> https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/README.build
> https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/descriptors/linux/gitian-tor.yml
> https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/descriptors/linux/gitian-firefox.yml
> https://gitweb.torproject.org/builders/tor-browser-bundle.git/blob/HEAD:/gitian/descriptors/linux/gitian-bundle.yml
> So far, the most progress I have made on #9444 is to do a reproducible
> build of the vanilla bundle.

I'm sure we'll make progress on this as I get up to speed. I added
myself to the CC list of #9444.

-Kevin

[1] https://trac.torproject.org/projects/tor/ticket/9939
[2] https://trac.torproject.org/projects/tor/ticket/9940
[3] https://trac.torproject.org/projects/tor/ticket/9941
[4] http://www.apple.com/legal/sla/docs/macosx107.pdf