[tor-bugs] #26578 [Core Tor/Tor]: Do clients request new consensus documents more often than we expect?

Sun Jul 1 14:08:43 UTC 2018

#26578: Do clients request new consensus documents more often than we expect?
--------------------------+------------------------
 Reporter:  arma          |          Owner:  (none)
     Type:  defect        |         Status:  new
 Priority:  Medium        |      Milestone:
Component:  Core Tor/Tor  |        Version:
 Severity:  Normal        |     Resolution:
 Keywords:                |  Actual Points:
Parent ID:                |         Points:
 Reviewer:                |        Sponsor:
--------------------------+------------------------

Comment (by teor):

 Replying to [ticket:26578 arma]:
 > In our user count estimates, we used the reasoning that clients fetch a
 new consensus document every 2 to 4 hours, or on average 3 hours, so that
 represents 8 fetches per day on average.

 That calculation is intuitively wrong: if consensuses are only valid for 3
 hours, then clients can't wait 4 hours to get a new one. As a first
 approximation, clients wait between 1 and 2 hours, and therefore they
 retry every 90 minutes.

 dir-spec gives a precise example:
     [For example, if a client has a consensus that became valid at 1:00,
     and is fresh until 2:00, and expires at 4:00, that client will fetch
     a new consensus at a random time between 2:45 and 3:50, since 3/4
     of the one-hour interval is 45 minutes, and 7/8 of the remaining 75
     minutes is 65 minutes.]
 https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3552

 But that's not enough to tell us how long clients wait *between*
 consensuses.

 The consensus that the cache has is also relevant:
     [For example, if a cache has a consensus that became valid at 1:00,
     and is fresh until 2:00, that cache will fetch a new consensus at
     a random time between 2:00 and 2:30.]
 https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n3367

 So here are the details of a steady-state operation:
 * from 02:00 to 03:00, all directory authorities have a consensus that
 became valid at 02:00, and this pattern continues every hour
   * so directory authorities always have a fresh consensus
 * any directory cache that has a consensus that became valid at 01:00, and
 is fresh until 02:00, will fetch a consensus that is fresh until 03:00
 sometime between 02:00 and 02:30, and this pattern continues every hour
   * so between 02:00 and 02:30, all caches either have an 01:00 or an
 02:00 consensus, and from 02:30 to 03:00 they have an 02:00 consensus
 * any client that has a consensus that became valid at 01:00, fresh until
 02:00, and valid until 04:00, will fetch a new consensus between 02:45 and
 03:50, and this pattern continues differently depending on exactly which
 consensus they fetch
   * here are the cases:
     * clients (15/65) that fetch between 02:45 and 03:00 get an 02:00
 consensus
     * half (15/65) the clients that fetch between 03:00 and 03:30 get an
 02:00 consensus, and half (15/65) get an 03:00 consensus, with the earlier
 clients more likely to get an 02:00 consensus
     * clients (20/65) that fetch between 03:30 and 03:50 get an 03:00
 consensus
   * so about half the clients:
     * fetch between 02:45 and 03:30, and get an 02:00 consensus, and retry
 between 3:45 and 4:50, a 15 minute to 125 minute interval, skewed towards
 125 minutes
     * fetch between 03:00 and 03:50 and get an 03:00 consensus, and retry
 between 4:45 and 5:50, a 55 minute to 170 minute interval, skewed towards
 55 minutes
   * overall, that's approximately 93 minutes between client attempts (due
 to the skew away from the extremes, and the larger number of clients in
 the later interval)

 > But in reality, it seems that clients fetch consensus documents way more
 frequently than that: looking at just my local Tor client, I see
 >
 > Jun 28 21:11:52.190 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 > Jun 28 22:43:52.355 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 92
 > Jun 28 23:59:52.417 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 76
 > Jun 29 01:42:52.501 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 103
 > Jun 29 03:33:52.601 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 111
 > Jun 29 05:09:52.699 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 96
 > Jun 29 06:04:52.754 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 115
 > Jun 29 07:54:52.874 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 110
 > Jun 29 08:56:52.946 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 62
 > Jun 29 10:32:53.036 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 96
 > Jun 29 12:36:53.121 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 124
 > Jun 29 14:06:53.186 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 90
 > Jun 29 14:53:53.215 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 47
 > Jun 29 15:52:53.256 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 59
 > Jun 29 17:15:53.319 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 83
 > Jun 29 18:20:53.367 [info] update_consensus_networkstatus_downloads():
 Launching microdesc standard networkstatus consensus download.
 65
 >
 >
 > So first, this means maybe our user counting algorithms are off, since
 they involve heuristics like "divide by 10 where 10 is an estimate of the
 average daily consensus fetches from a client."

 24*60/93 = 15.48

 > And second, does it mean that we are putting more load on the network
 than we expected, or need? How often do clients need a new consensus
 document really?

 Well, we could easily make the consensus valid for another few hours.
 Making it fresh for longer probably won't help, because it delays
 directory mirror consensus updates as well.

 Let's make a spreadsheet and tweak the numbers?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/26578#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online