How to Run High Capacity Tor Relays

[coderman]

On Tue, Aug 24, 2010 at 8:27 AM, Mike Perry <mikeperry at fscked.org> wrote:
> ...
> # Set the hard limit of open file descriptors really high.
> # Tor will also potentially run out of ports.
> ulimit -SHn 65000

typically in /etc/security/limits.conf. i like to append:
*               soft    nofile          4096
*               hard    nofile          65535

but on big servers use .25mm as hard limit. (Tor not this fd hungry,
64k is fine)


> # Load an amalgam of gigabit-tuning sysctls from:
> ...
> # We have no idea which of these are needed yet for our actual use
> # case, but they do help (especially the nf-contrack ones):

you probably want to save in /etc/sysctl.conf , then sysctl -p


> ...
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 65536 16777216
> net.core.netdev_max_backlog = 2500
> net.ipv4.tcp_no_metrics_save = 1
> net.ipv4.tcp_moderate_rcvbuf = 1
> net.core.rmem_max = 1048575
> net.core.wmem_max = 1048575

^- these are important and useful



> net.ipv4.ip_local_port_range = 1025 61000

^- that's a little aggressive, better to set FIN timeout lower. i like
5000 to 65535 ephemeral port range


> net.ipv4.tcp_max_syn_backlog = 10240
> net.ipv4.tcp_fin_timeout = 30

^- i like a fin timeout of 3-4 seconds on a busy server, otherwise
you've got lots of resources tied up in sockets waiting to die...  Tor
not quite so volatile as some services, so perhaps 30 is fine.


> net.ipv4.tcp_keepalive_time = 1200

^- who uses keepalive? :)


> net.netfilter.nf_conntrack_tcp_timeout_established=7200
> net.netfilter.nf_conntrack_checksum=0
> net.netfilter.nf_conntrack_max=131072
> net.netfilter.nf_conntrack_tcp_timeout_syn_sent=15

^- best to just disable conntrack altogether if you can. -J NOTRACK in
the raw table as appropriate.
you're going to each up lots of memory with a decent nf|ip_conntrack_max
( check /proc/sys/net/ipv4/netfilter/ip_conntrack_max , etc )


> [...]
some dupes in here?

> net.ipv4.ip_forward=1
> ...
> net.ipv4.conf.default.forwarding=1
> net.ipv4.conf.default.proxy_arp = 1

^- BAD! this should not be enabled by default unless you're actually
routing specifically to guest vm's or between interfaces or something.
if you enable forwarding by default, someone may use you to relay some
malicious traffic.

were these cut and paste errors?  remember to disable forwarding
first, before tuning other parameters, as changing this value will
reset some others back to defaults. (!!)


> net.ipv4.tcp_syncookies = 1

^- not usually worth the overhead?


> net.ipv4.conf.all.rp_filter = 1

^- note that you need to be precise with your routing metrics and such
for multi-homed with rp_filter enabled. also, this costs resources,
and if you can avoid it, do so.


> net.ipv4.conf.default.send_redirects = 1
> net.ipv4.conf.all.send_redirects = 0

^- don't know if these are too useful either. i prefer to limit ICMP
beyond this. (perhaps related to forwarding defaults above.) Ex:
echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_all
echo "0" > /proc/sys/net/ipv4/conf/all/accept_redirects
echo "1" > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses





> == Did I leave anything out? ==
>
> Well, did I?

i'd love to see an sca6000 accelerated node.  been working with these
recently but unfortunately they're allocated for other work...
(most of the other crypto hw is going to be bus / implementation
limited to less than what a beefy 64bit modern server can provide, so
of little utility in this context.)

best regards,
***********************************************************************
To unsubscribe, send an e-mail to majordomo at torproject.org with
unsubscribe or-talk    in the body. http://archives.seul.org/or/talk/