[tor-talk] Hidden Services - reliably resolving/using onions, CPU

[grarpamp]

When trying to fetch about 50 onions at a time, Tor eats up all of
a P4 1.8GHz CPU. And only about a tenth of the HTTP requests that
are expected to resolve and return data ever do return said data.
Further, using Tor as an exit (whether via polipo or SOCKS) also
becomes less than useful.

I'm not seeing any kernel, Tor, or polipo errors.
Polipo uses: clientTimeout=380 serverIdleTimeout=360 serverTimeout=360.
Bandwidth is around 200k and not a problem.
Wget uses: -t 0 --retry-connrefused --max-redirect 0.
Wget does return these log counts across a couple thousand unique onions:
 - The non SOCKS ones are known and ok.
 - I'm not sure what the SOCKS ones mean in regards to Tor?

   4 0 redirections exceeded.
  49 ERROR 403: Forbidden port.
   1 ERROR 500: Internal Server Error.
   2 ERROR 502: Server dropped connection.
  58 ERROR 504: Connect to onion failed: General SOCKS server failure.
   5 ERROR 504: Connect to onion failed: SOCKS connection not allowed.
2310 ERROR 504: Connect to onion failed: SOCKS error: TTL expired.
   1 ERROR 504: Connect to onion failed: SOCKS error: connection refused.
 111 ERROR 504: Connect to onion failed: SOCKS error: host unreachable.

When I cut it back to 10 in parallel, I have free CPU and better
results (and shifts TTL's to unreach as well).

  24 0 redirections exceeded.
   4 Authorization failed.
   3 Cookie coming from onion attempted to set domain to onion
   3 Cookie coming from onion attempted to set domain to onion
  36 ERROR 403: Forbidden port.
   1 ERROR 403: Forbidden.
   3 ERROR 404: Not Found.
   1 ERROR 500: Internal Server Error.
   1 ERROR 502: Bad Gateway.
   1 ERROR 502: Couldn't parse server status line.
   1 ERROR 502: Read from server failed: Timeout.
  20 ERROR 502: Server dropped connection.
   1 ERROR 503: Service unavailable.
   6 ERROR 504: Connect to onion failed: General SOCKS server failure.
  37 ERROR 504: Connect to onion failed: SOCKS connection not allowed.
 108 ERROR 504: Connect to onion failed: SOCKS error: TTL expired.
   6 ERROR 504: Connect to onion failed: SOCKS error: connection refused.
1410 ERROR 504: Connect to onion failed: SOCKS error: host unreachable.
   1 No data received.

I'm not sure yet whether it's the parallelism, lack of CPU, or
something else that's causing it (I don't have enough free CPU to
test higher levels of parallelism).

Overall, it seems Tor acts unreliably when loaded down with hidden
service requests. Whereas with TCP over a loaded internet or CPU,
you normally get connections and data, just more slowly. This
behavior with Tor could very easily impact chatty mesh services
that people might want to deploy on onions. Ideas?

(I didn't test whether the same thing happens with paralellizing
clearnet sites via Tor's exit feature. Or with tons of parallel
connections to just a few onions.)