[tor-bugs] #25687 [Core Tor/Tor]: over-report of observed / self-measure bandwidth on fast hardware -- important to torflow / peerflow

[Tor Bug Tracker & Wiki]

#25687: over-report of observed / self-measure bandwidth  on fast hardware --
important to torflow / peerflow
-------------------------------------------------+-------------------------
 Reporter:  starlight                            |          Owner:  (none)
     Type:  defect                               |         Status:  new
 Priority:  Medium                               |      Milestone:  Tor:
                                                 |  unspecified
Component:  Core Tor/Tor                         |        Version:  Tor:
                                                 |  0.2.6.10
 Severity:  Normal                               |     Resolution:
 Keywords:  tor-bwauth, needs-research, needs-   |  Actual Points:
  proposal?                                      |
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by teor):

 Replying to [comment:13 starlight]:
 > The essence of Torflow's active approach is that observed bandwidth
 capacity at each relay is the key measurement and that it can only be
 reliably determined locally but that it requires adjustment, principally
 to account for used vs unused capacity and secondly the relative
 performance of each node in the asymmetric domain of internet traffic
 routing.  IMO indisputably correct.  The Peerflow paper tacitly recognizes
 this.

 Our parent ticket for these kinds of adjustments is #27346.

 > However the simple linear adjustment algorithm cannot be fine-tuned for
 better results across the vast range of relay performance.  IIRC
 polynomial equations of sufficient order can describe curves of near
 arbitrary complexity and therefore parameterized polynomials can be used
 interactively, in a gradual empirical search, to describe an improving set
 of adjustment biases for applying scanner measurements to advertised
 bandwidths.  This link illustrates the general principal, though the idea
 is to design and construct bias curves with polynomials rather then to fit
 them somehow.
 >
 > https://en.wikipedia.org/wiki/Polynomial_regression

 I have opened #27790 for this idea.

 > Averaging all relay measurements to a single value appears too simple in
 the face of reality.  My suggestion was and is to apply some variation of
 a moving average in such a way that relay biases are determined relative
 to relays of similar capacity rather than to all relays.  Note this is not
 at all the same as the "slice" scheme used by Torflow to group relays for
 measurement and I agree eliminating it was a good idea.
 >
 > https://en.wikipedia.org/wiki/Moving_average

 The ticket for implementing moving averages is #27789.

 > Evaluating relays against others of the same class seems a good idea.
 (I.e. process guards, exits and unflagged middle relays as isolated
 collections.)  Torflow implemented it when PID logic was active, but
 presently it is disabled.

 I opened #27791for this idea.

 > I believe the "filtered" and "stream" logic in Torflow increases noise
 in the system rather than reducing it, and functions in opposition to the
 intent of the design.  The "filtered" treatment option should be
 eliminated.

 We have no plans to implement this feature in sbws. We have no plans to
 modify torflow.

 > Central to all the above is iterative empirical adjustment possibly
 assisted by automated collection and analysis, with control knobs exposed
 as consensus parameters.  The above can easily be implemented in a manner
 that allows the new system to start exactly where Torflow is and to
 cautiously refine, with the option to reverse course at any point.

 There are quite a few tickets open with metrics, torflow, and sbws for
 this kind of analysis. For example, see #7177, #24045, #24834, #25459,
 #21994.

 The existing performance and bandwidth distribution metrics are quite
 good, too:
 https://metrics.torproject.org/torperf.html
 https://metrics.torproject.org/rs.html#map

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25687#comment:14>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online