[tor-dev] Help with rebasing arma's n23-5 tor branch to current master

[Karsten Loesing]

Hi Charlie,

did you have the chance to simulate your rebased n23-5 branch using Shadow?

If not, maybe I can help by simulating it once and writing down more
specific instructions for you?

Thanks!
Karsten


On 8/8/13 10:36 AM, Karsten Loesing wrote:
> (Moving this discussion back to tor-dev@ with Charlie's permission.)
> 
> On 8/6/13 4:10 AM, Charlie Belmer wrote:
>> Hey Karsten,
>>
>> I've been reading up on Shadow & Scallion to do some performance testing on
>> N23. Before I go too far down that road, I wanted to check in with you.
>>
>> Are there existing blueprints or best practices I should be following?
> 
> Thanks for looking into this!
> 
> So, I wouldn't call the following notes blueprints or best practices,
> but they are a description of what I would do to run Shadow simulations
> of your branch.
> 
> First of all, be sure to read the Shadow wiki which has contains lots of
> interesting stuff: https://github.com/shadow/shadow/wiki
> 
> The next step would be to get a vanilla Tor master (that your branch is
> based on) running in a minimal or tiny Shadow network.  It's probably
> easiest to use a local Ubuntu VM for this.  Or you can start an m1.large
> EC2 instance for the tiny network, or an even smaller instance for the
> minimal network.
> 
> Once that simulation succeeds, you'll want to run another simulation
> using your branch, extract performance metrics using Shadow's analyze
> script, produce a PDF with graphs also using the analyze script, and
> then wonder if things are faster than before.
> 
> The minimal or tiny networks are mostly there to learn that things are
> working as expected.  You'll probably want larger networks for real
> performance results, but starting small to get everything working is
> probably easier.
> 
> Sounds doable, I guess.  However, here's where things start to get
> tricky: in order to simulate Tor master (or a branch based on it) in
> Shadow, you'll have to perform quite a few extra steps like compiling
> your own CMake and Clang/LLVM.  Not really hard, but you need to get all
> pieces together before Shadow does what you want.
> 
> I'm pasting detailed instructions that I used to simulate Steven's uTP
> branch in Shadow below as an example.  Not everything applies to your
> branch, but it should give you the general idea.
> 
> If something remains unclear, please feel free to ask!
> 
>> Depending how it goes, I'd also look at re-trying adaptive 23 or tuning the
>> existing implementation.
> 
> Neat!  We should probably include Mashael in that discussion once you
> have a working and simulated n23-5 branch.  Maybe she has ideas what
> parts need tweaking.  I'm cc'ing her in this mail so she knows what
> you're working on.
> 
> In any case, having a working rebased n23-5 branch is already a big step
> forward.
> 
> Thanks!
> Karsten
> 
> 
> === START OF INSTRUCTIONS ===
> 
> Launch new EC2 instance running Ubuntu Server 13.04 64 bit
> 
> Set ulimit -n to 25000, see Shadow wiki for details
> https://github.com/shadow/shadow/wiki/Preparing-your-machine#system-configs-and-limits
> 
> $ sudo apt-get update
> $ sudo apt-get upgrade
> 
> Clone and test-compile modified libutp
> 
> $ sudo apt-get install build-essential git automake libssl-dev libevent-dev
> $ mkdir src
> $ cd src/
> $ git clone https://github.com/sjmurdoch/libutp
> $ cd libutp/
> $ make
> $ cd ../
> 
> Clone and test-compile Tor branch utp
> 
> $ git clone https://git.torproject.org/tor.git
> $ cd tor/
> $ git remote add sjm217 https://git.torproject.org/sjm217/tor.git
> $ git fetch sjm217
> $ git checkout -b utp sjm217/utp
> $ ./autogen.sh
> $ LDFLAGS="-L/home/ubuntu/src/libutp" CFLAGS="-I/home/ubuntu/src/libutp"
> LIBS="-lutp -lrt" ./configure --disable-asciidoc
> $ make
> $ make distclean
> $ cd ../
> 
> Modify Tor to use uTP for all links (or "0 &&" for not using uTP for any
> link)
> 
> diff --git a/src/or/channeltls.c b/src/or/channeltls.c
> index 0551b73..b7b36e1 100644
> --- a/src/or/channeltls.c
> +++ b/src/or/channeltls.c
> @@ -418,7 +418,7 @@ channel_tls_connect(const tor_addr_t *addr, uint16_t
> port,
>    /* Create a uTP connection */
>    tor_addr_to_sockaddr(addr, port, (struct sockaddr*)&sin, sizeof(sin));
>    tor_addr_to_str(addr_str, addr, sizeof(addr_str), 0);
> -  if (!strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) {
> +  if (1 || !strncmp(addr_str, "128.232.10.129", sizeof(addr_str))) {
>      log_info(LD_CHANNEL,
>               "Trying uTP connection to %s", addr_str);
>      tlschan->utp = UTP_Create(tor_UTPSendToProc, tlschan, (const struct
> sockaddr*)&sin,
> 
> Install CMake 2.8.10
> 
> $ wget http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz
> $ tar xf cmake-2.8.10.2.tar.gz
> $ cd cmake-2.8.10.2/
> $ ./configure
> $ make
> $ export PATH=~/src/cmake-2.8.10.2/bin:$PATH
> $ cd ../
> $ which cmake # result: /home/ubuntu/src/cmake-2.8.10.2/bin/cmake
> 
> Build Clang and LLVM from source
> 
> $ wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
> $ wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz
> $ tar xaf llvm-3.2.src.tar.gz
> $ tar xaf clang-3.2.src.tar.gz
> $ cp -R clang-3.2.src llvm-3.2.src/tools/clang
> $ cd llvm-3.2.src/
> $ mkdir build
> $ cd build/
> $ cmake -DCMAKE_INSTALL_PREFIX=/home/ubuntu/.local ../.
> $ make
> $ make install
> $ export PATH=~/.local/bin/:$PATH
> $ cd ../../
> $ which clang # result: /home/ubuntu/.local/bin//clang
> 
> Install Shadow
> 
> $ sudo apt-get install gcc xz-utils make automake autoconf cmake tidy
> libtidy-dev libglib2.0 libglib2.0-dev dstat pdftk python2.7
> python-matplotlib python-numpy python-scipy
> $ git clone https://github.com/shadow/shadow.git
> $ cd shadow
> $ ./setup dependencies
> $ ./setup build -g -i /home/ubuntu/src/libutp -i
> /usr/include/x86_64-linux-gnu/c++/4.7 -l /home/ubuntu/src/libutp
> --tor-prefix /home/ubuntu/src/tor --tor-lib utp
> $ ./setup install
> $ export PATH=~/.shadow/bin/:$PATH
> 
> Simulate tiny network in Shadow
> 
> $ cd resource/examples/scallion/
> $ tar xf tiny-4GB-m1.large.tar.xz
> $ cd tiny-4GB-m1.large/
> $ scallion
> 
> Graph simulation results
> 
> Add one line to beginning of analyze.py:
> 
> import matplotlib; matplotlib.use('Agg')
> 
> For additional traffic simulations:
> 
> $ grep -v "\[traffic-" scallion.log > scallion-notraffic.log
> 
> $ cd ~/src/
> $ mkdir results
> $ cd results/
> $ python ~/src/shadow/contrib/analyze.py parse --output uTP-1
> ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-1/scallion.log
> $ python ~/src/shadow/contrib/analyze.py parse --output uTP-0
> ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-utp-0/scallion.log
> $ python ~/src/shadow/contrib/analyze.py parse --output vanilla
> ~/src/shadow/resource/examples/scallion/tiny-4GB-m1.large/data-vanilla/scallion.log
> $ python ~/src/shadow/contrib/analyze.py plot --title "sjm217's utp
> branch (#9166)" --prefix "sjm217-utp" --data vanilla/ "vanilla
> 0.2.4.4-alpha" --data uTP-1/ "uTP for all links" --data uTP-0/ "uTP for
> none of the links"
> 
> === END OF INSTRUCTIONS ===
> 
>