Proposal 168: Reduce default circuit window

Fri Sep 11 17:50:30 UTC 2009

Hello Karsten

I have to agree with Björn, reducing the circuit window parameter is a
bad idea (we are using a flow control mechanism to try to solve
congestion control).  I believe that Björn's student simulations would
be true. Since most websites nowadays are larger than 50KB the user
experience at the end of the day would be even worse (slower and
burstier experience) but also the network would be internaly bw limited
to about 400 kbps (assuming 125ms between routers and the client and
only taking into account TCP dequeing,) even with infinite bandwidth
between nodes. Assuming some large values for throgput and very small
queues the througput for each circuit is limited to 190-129 kbps
depending on assumptions.

I think that would really reduce the ability to use the Tor network as
default in a daily basis.

Camilo

Assumptions for the values above:
avg RTT between nodes=125ms
avg ckt per node =10
Average throughout of the slowest node in a circuit = 500 KB/s
Window size =50KB
Average Tor queue size= 100KB

Simple table for the Througput limits.
               Latency(s)    Throughput (KB/s)    Througput (kbps)
Just natural     0.375        133.33                1066.67        
*infinite throughput
+TCP queues      1.000        50                     400           
*infinite throughput
+ trasmission    1.100        45.45                  363.64        
->slowest link is 500KB and only dedicates to the connections!
+ Tor queueing   2.100        23.81                  190.48         ->
as above with queing 100KB on average for the nodes.

Karsten Loesing wrote:
> Hi Björn,
>
> I'm commenting on just a few points here:
>
> On 09/01/2009 01:50 PM, Björn Scheuermann wrote:
> > this sounds like an interesting proposal! However, I'd like to point
> you to
> > some implications and side effects. Over the past months, one of my
> students
> > (Daniel Marks, on CC) has worked on congestion control in Tor. One
> of his
> > results is an implementation of Tor circuits in a network simulation
> framework
> > (ns-2). We use this implementation to simulate Tor networks of
> different size
> > and with different traffic patterns, play with various parameters
> and measure
> > throughput, cell latencies, etc. Currently, our aim is to gain a better
> > understanding of congestion effects in Tor and their implications and
> > interrelations.
>
> Sounds great. Are you planning to release your simulation sources? I'd
> lie when saying that I want to look at the sources now, but I certainly
> would like to play with your simulation at a later point. As may others.
>
> >> 2. Motivation
> >>
> >>   Karsten's torperf graphs show that the median download time for a
> 50KB
> >>   file over Tor in mid 2009 is 7.7 seconds, whereas the median download
> >>   time for 1MB and 5MB are around 50s and 150s respectively. The 7.7
> >>   second figure is way too high, whereas the 50s and 150s figures are
> >>   surprisingly low.
> > Your statement implies that you are looking at something like the
> > "transmission latency per KB" (which is, by the way, identical to
> the inverse
> > throughput).
>
> True, most of the torperf graphs are for measuring bandwidth rather than
> latency. See the PDF reports for more information:
>
> https://git.torproject.org/checkout/metrics/master/report/performance/torperf-2009-08-24.pdf
>
> https://git.torproject.org/checkout/metrics/master/report/circwindow/circwindow-2009-08-19.pdf
>
> The early results of cells in circuit queues might help, too. Note that
> more work remains to make real use of these data. Want to help? :)
>
> https://git.torproject.org/checkout/metrics/master/report/buffer/bufferstats-2009-08-25.pdf
>
> >>   To get a more concrete sense of the benefit, though, Karsten has been
> >>   running torperf side-by-side on exit relays with the old package
> window
> >>   vs the new one. The results are mixed currently -- it is slightly
> faster
> >>   for fetching 40KB files, and slightly slower for fetching 50KB files.
> > Note that file sizes of 40 or 50 KB cannot tell you much about such a
> > modification. As you state above, 100 cells is equivalent to ~50 KB
> of payload.
> > I.e., you do the whole transmission within the initial transfer
> window. So,
> > the data transfer will be over before the circuit's window
> mechanisms have had
> > a chance to become active at all! Consequently, modifications to the
> window
> > mechanism will not impact transfers that are below that threshold
> directly.
>
> The 50 KiB (kibibytes, not kilobytes) download should not fit into one
> circuit window of 101 cells. That circuit window can hold up to 101 x
> 498 = 50298 bytes whereas the file size is 51200 bytes. Due to the
> torperf output, the client receives even 51466 bytes. So, we need a
> second circuit window to transfer the 50 KiB file.
>
> But I admit that 50 KiB is still pretty low. Right now, we have more
> measurements running with 1 MiB files. They just take somewhat longer to
> complete, because we cannot start new downloads in 5-minute intervals,
> but only in 30-minute intervals. At the moment we have 347 data points;
> not enough to make a useful statement. It's going to take another two
> weeks to have enough data.
>
> >>   I think it's going to be tough to get a clear conclusion that this is
> >>   a good design just by comparing one exit relay running the patch. The
> >>   trouble is that the other hops in the circuits are still getting
> bogged
> >>   down by other clients introducing too much traffic into the network.
> > Maybe I'm getting something royally wrong - but what exactly are the
> > mechanisms within an onion router that would result in one circuit
> > experiencing much longer cell queuing delays just because *another*
> circuit
> > has a long queue in that router? In terms of fair resource sharing
> between
> > circuits (and the opportunity to forward cells quickly), our
> impression is
> > that the round-robin scheduler does its job pretty well. It will be
> hard to
> > improve on that just by introducing intentional resource
> underutilization.
> > But you guys know the Tor sources much better than we do - are we
> missing
> > something important here?
>
> The problem of loud circuits slowing down other circuits emerges when
> these circuits are sharing the same TCP connection.
>
> Did you have a look at Joel's and Ian's USENIX paper?
>
> "All traffic between any pair of routers, even if they represent
> circuits for different clients, are multiplexed over a single TCP
> connection. This results in interference across circuits during
> congestion control, packet dropping and packet reordering. This
> interference greatly contributes to Tors notorious latency problems."
>
> See http://crysp.uwaterloo.ca/publications/ for the publication list or
> http://www.cypherpunks.ca/~iang/pubs/TorTP.pdf for the PDF.
>
> Best,
> --Karsten
>