[tor-dev] Help with rebasing arma's n23-5 tor branch to current master

Thu Aug 8 08:36:45 UTC 2013

(Moving this discussion back to tor-dev@ with Charlie's permission.)

On 8/6/13 4:10 AM, Charlie Belmer wrote:
> Hey Karsten,
> 
> I've been reading up on Shadow & Scallion to do some performance testing on
> N23. Before I go too far down that road, I wanted to check in with you.
> 
> Are there existing blueprints or best practices I should be following?

Thanks for looking into this!

So, I wouldn't call the following notes blueprints or best practices,
but they are a description of what I would do to run Shadow simulations
of your branch.

First of all, be sure to read the Shadow wiki which has contains lots of
interesting stuff: https://github.com/shadow/shadow/wiki

The next step would be to get a vanilla Tor master (that your branch is
based on) running in a minimal or tiny Shadow network.  It's probably
easiest to use a local Ubuntu VM for this.  Or you can start an m1.large
EC2 instance for the tiny network, or an even smaller instance for the
minimal network.

Once that simulation succeeds, you'll want to run another simulation
using your branch, extract performance metrics using Shadow's analyze
script, produce a PDF with graphs also using the analyze script, and
then wonder if things are faster than before.

The minimal or tiny networks are mostly there to learn that things are
working as expected.  You'll probably want larger networks for real
performance results, but starting small to get everything working is
probably easier.

Sounds doable, I guess.  However, here's where things start to get
tricky: in order to simulate Tor master (or a branch based on it) in
Shadow, you'll have to perform quite a few extra steps like compiling
your own CMake and Clang/LLVM.  Not really hard, but you need to get all
pieces together before Shadow does what you want.

I'm pasting detailed instructions that I used to simulate Steven's uTP
branch in Shadow below as an example.  Not everything applies to your
branch, but it should give you the general idea.

If something remains unclear, please feel free to ask!

> Depending how it goes, I'd also look at re-trying adaptive 23 or tuning the
> existing implementation.

Neat!  We should probably include Mashael in that discussion once you
have a working and simulated n23-5 branch.  Maybe she has ideas what
parts need tweaking.  I'm cc'ing her in this mail so she knows what
you're working on.

In any case, having a working rebased n23-5 branch is already a big step
forward.

Thanks!
Karsten


=== START OF INSTRUCTIONS ===

Launch new EC2 instance running Ubuntu Server 13.04 64 bit

Set ulimit -n to 25000, see Shadow wiki for details
https://github.com/shadow/shadow/wiki/Preparing-your-machine#system-configs-and-limits

$ sudo apt-get update
$ sudo apt-get upgrade

Clone and test-compile modified libutp

$ sudo apt-get install build-essential git automake libssl-dev libevent-dev
$ mkdir src
$ cd src/
$ git clone https://github.com/sjmurdoch/libutp
$ cd libutp/
$ make
$ cd ../

Clone and test-compile Tor branch utp

$ git clone https://git.torproject.org/tor.git
$ cd tor/
$ git remote add sjm217 https://git.torproject.org/sjm217/tor.git
$ git fetch sjm217
$ git checkout -b utp sjm217/utp
$ ./autogen.sh
$ LDFLAGS="-L/home/ubuntu/src/libutp" CFLAGS="-I/home/ubuntu/src/libutp"
LIBS="-lutp -lrt" ./configure --disable-asciidoc
$ make
$ make distclean
$ cd ../

Modify Tor to use uTP for all links (or "0 &&" for not using uTP for any
link)

diff --git a/src/or/channeltls.c b/src/or/channeltls.c
index 0551b73..b7b36e1 100644
--- a/src/or/channeltls.c
+++ b/src/or/channeltls.c
@@ -418,7 +418,7 @@ channel_tls_connect(const tor_addr_t *addr, uint16_t
port,
   /* Create a uTP connection */
   tor_addr_to_sockaddr(addr, port, (struct sockaddr*)&sin, sizeof(sin));
   tor_addr_to_str(addr_str, addr, sizeof(addr_str), 0);
-  if (!strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) {
+  if (1 || !strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) {
     log_info(LD_CHANNEL,
              "Trying uTP connection to %s", addr_str);
     tlschan->utp = UTP_Create(tor_UTPSendToProc, tlschan, (const struct
sockaddr*)&sin,

Install CMake 2.8.10

$ wget http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz
$ tar xf cmake-2.8.10.2.tar.gz
$ cd cmake-2.8.10.2/
$ ./configure
$ make
$ export PATH=~/src/cmake-2.8.10.2/bin:$PATH
$ cd ../
$ which cmake # result: /home/ubuntu/src/cmake-2.8.10.2/bin/cmake

Build Clang and LLVM from source

$ wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
$ wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz
$ tar xaf llvm-3.2.src.tar.gz
$ tar xaf clang-3.2.src.tar.gz
$ cp -R clang-3.2.src llvm-3.2.src/tools/clang
$ cd llvm-3.2.src/
$ mkdir build
$ cd build/
$ cmake -DCMAKE_INSTALL_PREFIX=/home/ubuntu/.local ../.
$ make
$ make install
$ export PATH=~/.local/bin/:$PATH
$ cd ../../
$ which clang # result: /home/ubuntu/.local/bin//clang

Install Shadow

$ sudo apt-get install gcc xz-utils make automake autoconf cmake tidy
libtidy-dev libglib2.0 libglib2.0-dev dstat pdftk python2.7
python-matplotlib python-numpy python-scipy
$ git clone https://github.com/shadow/shadow.git
$ cd shadow
$ ./setup dependencies
$ ./setup build -g -i /home/ubuntu/src/libutp -i
/usr/include/x86_64-linux-gnu/c++/4.7 -l /home/ubuntu/src/libutp
--tor-prefix /home/ubuntu/src/tor --tor-lib utp
$ ./setup install
$ export PATH=~/.shadow/bin/:$PATH

Simulate tiny network in Shadow

$ cd resource/examples/scallion/
$ tar xf tiny-4GB-m1.large.tar.xz
$ cd tiny-4GB-m1.large/
$ scallion

Graph simulation results

Add one line to beginning of analyze.py:

import matplotlib; matplotlib.use('Agg')

For additional traffic simulations:

$ grep -v "\[traffic-" scallion.log > scallion-notraffic.log

$ cd ~/src/
$ mkdir results
$ cd results/
$ python ~/src/shadow/contrib/analyze.py parse --output uTP-1
~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-1/scallion.log
$ python ~/src/shadow/contrib/analyze.py parse --output uTP-0
~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-0/scallion.log
$ python ~/src/shadow/contrib/analyze.py parse --output vanilla
~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-vanilla/scallion.log
$ python ~/src/shadow/contrib/analyze.py plot --title "sjm217's utp
branch (#9166)" --prefix "sjm217-utp" --data vanilla/ "vanilla
0.2.4.4-alpha" --data uTP-1/ "uTP for all links" --data uTP-0/ "uTP for
none of the links"

=== END OF INSTRUCTIONS ===

