[tor-dev] Statistical inference in live network perf experiments

[Jansen, Robert G CIV USN NRL (5543) Washington DC (USA)]

Hi Mike,

So I just read your mail in the "Initial ideas from the culture/VPN thread" where you state:

> The core problem: People who do not have time to monitor the full workflow of other groups and get properly involved in them should not randomly interject to demand things that are already documented in tickets, public list posts, proposals, wiki pages, meetings, and irc discussion.

I feel like I have done exactly that in my previous mail below, and I sincerely apologize. I agree with you, random interjections from the uninformed likely cause more harm than good, so I should just stay out of it.

We _are_ working on the topic of statistical inference, but I'm sure you already have a solid plan, so please just ignore my noise.

Again, I am truly sorry, and I'll try not to make this mistake again.

Peace, love, and positivity,
Rob

> On Oct 17, 2020, at 7:43 PM, Jansen, Robert G CIV USN NRL (5543) Washington DC (USA) <rob.g.jansen at nrl.navy.mil> wrote:
> 
> Hello Mike and David!
> 
> I see that we are in the process of testing some changes to KIST parameters, and have more upcoming perf experiments planned. It will be great to gather additional information to help us better understand how to improve Tor performance :)
> 
> My main concern about running experiments on the live network is the lack of control [0] across the uncountable number of confounding variables present across the network.
> 
> Because of the lack of control, I strongly advise that statistical inference techniques are applied when conducting the experiments and analyzing the results. Ideally you will estimate and quantify sources of systematic error. Repeated trials and A/B testing (when that makes sense) will help increase our confidence in your results.
> 
> We have some recent work on these topics (in submission). DM if you are interested in a copy of the paper.
> 
> Peace, love, and positivity,
> Rob
> 
> [0] https://en.wikipedia.org/wiki/Scientific_control