[tor-dev] Statistical inference in live network perf experiments

[Jansen, Robert G CIV USN NRL (5543) Washington DC (USA)]

Hello Mike and David!

I see that we are in the process of testing some changes to KIST parameters, and have more upcoming perf experiments planned. It will be great to gather additional information to help us better understand how to improve Tor performance :)

My main concern about running experiments on the live network is the lack of control [0] across the uncountable number of confounding variables present across the network.

Because of the lack of control, I strongly advise that statistical inference techniques are applied when conducting the experiments and analyzing the results. Ideally you will estimate and quantify sources of systematic error. Repeated trials and A/B testing (when that makes sense) will help increase our confidence in your results.

We have some recent work on these topics (in submission). DM if you are interested in a copy of the paper.

Peace, love, and positivity,
Rob

[0] https://en.wikipedia.org/wiki/Scientific_control