[tor-relays] Rampup speed of Exit relay

[teor]

Hi,

I wanted to highlight the following parts of my reply:

The Tor network is quite happy to allocate around 47 MByte/s (yes, 376 MBit/s) to your relay based on its bandwidth measurements, but it won't do that until your relay shows it is actually capable of sustaining that traffic over a 10-second period (the observed bandwidth). At the moment, your relay can only do 19.83 MByte/s, so that's what it's being allocated.

Maybe your provider has good connectivity to the bandwidth authorities, but bad connectivity elsewhere?
Maybe your provider is otherwise limiting output traffic?

Do you run a local, caching, DNS resolver?
That could be a bottleneck, as almost all of your observed bandwidth is based on exit traffic.

The details are below:

> On 22 Sep 2016, at 03:58, D. S. Ljungmark <spider at takeit.se> wrote:
> 
> On tor, 2016-09-22 at 06:29 +1000, teor wrote:
>>> 
>>> On 22 Sep 2016, at 05:41, nusenu <nusenu at openmailbox.org> wrote:
>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> So, how do we get tor to move past 100-200Mbit? Is it just a
>>>>>> waiting game?
>>> 
>>> I'd say just run more instances if you still have resources left
>>> and
>>> want to contribute more bw.
>>> (obviously also your exit policy influences on how much your
>>> instance is
>>> used)
>> 
>> In my experience, a single Tor instance can use between 100Mbps and
>> 500Mbps.
>> It's highly dependent on the processor, OpenSSL version, and various
>> network factors.
> 
> Acknowledged, the question is, how do you measure that.

There's a tool called "chutney" that can calculate an effective local bandwidth based on how fast a Tor test network can transmit data.
Using it, I discovered that two machines that were meant to be almost identical had approximately 1.5 Gbps and 500 Mbps capacity, because one had a slightly older processor.

This is how I test the local CPU capacity for Tor's crypto:

git clone https://git.torproject.org/chutney.git
chutney/tools/test-network.sh --flavor basic-min --data 10000000

This is how I test the local file descriptor and other kernel data structure capacity:

chutney/tools/test-network.sh --flavor basic-025

This is how I test what a single client can push through a Tor Exit (I think Exits limit each client to a certain amount of bandwidth, but I'm not sure - maybe it's 1 Mbit/s or 1 MByte/s):

tor DataDirectory /tmp/tor.$$ PidFile /tmp/tor.$$/tor.pid SOCKSPort 2000 ExitNodes <your-exit-1>
sleep 10
curl --socks5-hostname 127.0.0.1:"$1" <a-large-download-url-here>


>> There are also some random factors in the bandwidth measurement
>> process, including the pair selection for bandwidth measurement. And
>> clients also choose paths randomly. This means some relays get more
>> bandwidth than others by chance.
> 
> That's interesting, how does the bandwidth scaling / metering work?
> Where/how does it decide how much bandwidth is available vs. what it
> announces to the world?

Tor has 5 bandwidth authorities that measure each relay around twice per week.
Then the median measurement is used for the consensus weight, which is the weight clients use to choose relays.
(The relay's observed bandwidth is also used to limit the consensus weight.)

In particular, your relay is currently limited by its observed bandwidth of 19.83 MByte/s.
Mouse over the "Advertised Bandwidth" figure to check:
https://atlas.torproject.org/#details/5989521A85C94EE101E88B8DB2E68321673F9405

The gory details of each bandwidth authority's vote are in:
https://collector.torproject.org/recent/relay-descriptors/votes/

Faravahar Bandwidth=10000 Measured=43800
gabelmoo Bandwidth=10000 Measured=78200
moria1 Bandwidth=10000 Measured=47200
maatuska Bandwidth=10000 Measured=40100
longclaw Bandwidth=10000 Measured=49500

(I usually look these up by hand, but I'm sure there's a faster way to do it using stem.)

So the Tor network is quite happy to allocate around 47 MByte/s (yes, 376 MBit/s) to your relay based on its bandwidth measurements, but it won't do that until your relay shows it is actually capable of sustaining that traffic over a 10-second period (the observed bandwidth). At the moment, your relay can only do 19.83 MByte/s, so that's what it's being allocated.

Maybe your provider has good connectivity to the bandwidth authorities, but bad connectivity elsewhere?
Maybe your provider is otherwise limiting output traffic?

Do you run a local, caching, DNS resolver?
That could be a bottleneck, as almost all of your observed bandwidth is based on exit traffic.

> Right now I can comfortable pull/push 700Mbit in either direction on
> this node, so that's where I left the setting, if there is a penalty
> for stating more bandwidth avialable than the network can measure, then
> I have a problem.

No, the network agrees with your setting - 376 MBit/s / 40% average Tor network utilisation is 940 Mbps.
(It's slightly higher than your actual bandwidth, possibly because it's an exit, or in a well-connected network position.)

>> If you want to use the full capacity of your Exit, run multiple Tor
>> instances.
> 
>> You can run 2 instances per IPv4 address, using different ports.
>> Many people choose ORPort 443 and DirPort 80 as their secondary
>> instance ports (this can help clients on networks that only allow
>> those ports), but you can choose any ports you want.
> 
> True, but there's a limit to how many nodes you can have in a /24, and
> I really want scaling up on a single node before adding more resources.

No there's not, not for a /24.
You can have 253 * 2 = 506 Tor instances in an IPv4 /24.

> Throw more resources at it cause we don't know why it sucks seems like
> such a devops thing to do.

Add another instance on the same machine.
Not more resources, but more processes, more threads using the existing resources.

>> Have you considered configuring an IPv6 ORPort as well?
>> (It's unlikely to affect traffic, but it will help clients that
>> prefer IPv6.)
> 
> Not sure right now, I've had _horrid_ experiences with running Tor on
> ipv6,  ranging from the absurd ( Needing ipv4 configured to set up
> ipv6)

IPv4 addresses are mandatory for relays for a few reasons:
* Tor assumes that relays form a fully-connected clique - this isn't possible if some are IPv4-only and some are IPv6-only;
* some of Tor's protocols only send an IPv4 address - they're being redesigned, but protocol upgrades are hard;
* until recently, clients could only bootstrap over IPv4 (and they still can't using microdescriptors, only full descriptors);
* and IPv6-only clients have poor anonymity, because they stick out too much.

> to the inane ( Config keys named "Port" not valid for both ipv6
> and ipv4, horrid documentation)

We're working on the IPv6 documentation, and happy to fix any issues.
What particular Port config?
What was bad about the documentation?

Feel free to log bugs against Core Tor/Tor, or we can log them for you:
https://trac.torproject.org/

> I've got quite a few ipv6 -only- networks, and I'd gladly put up a
> number of relays/Exits there ( using ephemeral addressing) however
> that's impossible last I tried it.

Yes, it is, see above for why.

> My general consensus from last I looked in depth at this is that "Tor
> doesn't support ipv6. It claims to, but it doesn't."

Choosing anonymous, random paths through a non-clique network (mixed IPv4-only, dual-stack, and IPv6-only) is an open research problem. We can't really implement IPv6-only relays until there are some reasonable solutions to this issue. Until then, we have dual-stack relays.

And IPv6-only Tor clients can connect using IPv6 bridges, or bootstrap over IPv6 with ClientUseIPv4 0 and UseMicrodescriptors 0.

That's pretty much the limit of what we can do with IPv6, until researchers come up with solutions to the non-clique issue.

>>>>> How long has the relay been up?
>>>> 
>>>> 4 years or so. ( current uptime: 11 hours since reboot, it
>>>> reboots weekly )
>>> 
>>> This relay (5989521A) has been first seen on 2014-04-10 according
>>> to
>>> https://onionoo.torproject.org (still long enough).
>>> 
>>> Why do you reboot weekly? Memory leak workaround?
>> 
>> If you reboot weekly, you will take time each week to re-gain
>> consensus weight and various other flags. For example, you will only
>> have the HSDir flag for ~50% of the time. (The Guard flag is also
>> affected, but it's somewhat irrelevant for Exits.)
> 
> Fancy that. "Don't upgrade your software because our software can't
> handle it" is one of those things that really bug me.

That's not what I said.
Upgrade your software. Have your relay go up and down as much as you like. The network will handle it fine. Tor clients will be fine.

But if you want to optimise your traffic, then fewer restarts are one thing to try. A restart per week certainly isn't typical of most Tor relays, so your relay looks less stable by comparison.

> How much downtime can the node have before losing consensus
> weight/flags?

A restart loses the HSDir flag for 72 hours, and the Guard flag for a period that is dependent on how old your relay is. (It should be inversely related, currently it seems to be positively correlated, which is a bug we're working on fixing.)

> Is it just for restarting the tor process as well?

Yes. Try sending it a HUP instead, when you've just changed the config.

Why do you (need to) restart weekly?

>> I'd avoid the reboots if you can, there's a known bug affecting at
>> least the Guard flag and restarts, where a long-lived stable relays
>> are disproportionately impacted compared with new relays. I haven't
>> seen any evidence that it affects other flags or consensus weight,
>> but you could try not restarting and see if that helps.
> 
> Right, I can tune that for a week and see.

Thanks. Hope it works out for you.

Tim

Tim Wilson-Brown (teor)

teor2345 at gmail dot com
PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
xmpp: teor at torproject dot org






-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20160922/251a1e17/attachment.sig>