[tor-relays] Metrics for assessing EFF's Tor relay challenge?

[Karsten Loesing]

On 27/03/14 11:57, Roger Dingledine wrote:
> Hi Christian, other tor relay fans,
> 
> I'm looking for some volunteers, hopefully including Christian, to work
> on metrics and visualization of impact from new relays.
> 
> We're working with EFF to do another "Tor relay challenge" [*], to both
> help raise awareness of the value of Tor, and encourage many people to
> run relays -- probably non-exit relays for the most part, since that's
> the easiest for normal volunteers to step up and do.
> 
> You can read about the first round from several years ago here:
> https://www.eff.org/torchallenge
> 
> To make it succeed, the challenge for us here is to figure out what to
> measure to track progress, and then measure it and graph it for everybody.
> 
> I'm figuring that like last time, EFF will collect a list of fingerprints
> of relays that signed up "because of the challenge".
> 
> One of the main pushes we're aiming for this year is longevity: it's
> easy to sign up a relay for two weeks and then stop. We want to emphasize
> consistency and encourage having the relays up for many months.

Before going through your list of things we'd want to track below, let's
first talk about our options to turn a list of fingerprints into fancy
graphs:

 1. Write a new metrics-web module and put graphs on the metrics
website.  This means parsing relay descriptors and storing certain
per-relay statistics for all relays.  That gives us maximum flexibility
in the kinds of statistics, but is also most expensive in terms of
developer hours.  I don't want to do this.

 2. Extend Globe to show details pages for multiple relays.  This
requires us to move to the server-based Globe-node, because the poor
browser shouldn't download graph data for all relays, but the server
should return a single graph for all relays.  It's also unclear if the
new graphs will be of general interest for Globe users, and if the rest
of the Globe details will be confusing to people interested in the relay
challenge.  Probably not a great idea, but I'm not sure.

 3. Extend Onionoo to return aggregate graph data for a given set of
fingerprints.  Seems useful.  But has the big disadvantage that Onionoo
would suddenly have to create responses dynamically.  I'm worried about
creating a new performance bottleneck there, and this is certainly not
possible with poor overloaded yatei.

 4. Write a new little tool that fetches Onionoo documents once (or
twice) per day for all relays participating in the relay challenge and
that produces graph data.  That new tool could probably re-use some
Compass code for the backend and some Globe code for the frontend.
Graphs could be integrated directly into EFF's website.  This is
currently my favorite approach.

Note for 2--4: Onionoo currently only gives out data for relays that
have been running in the past 7 days.  I'd have to extend it to give out
all data for a list of fingerprints, regardless of when relays were
running the last time.  That's 2--3 days of coding and testing for me.
It's also potentially creating a bottleneck, so we should first have a
replacement for yatei.

> So what are the things we'd want to track?
> 
> - Number of relays signed up that are Running, over time.

We can do something here with Onionoo's new uptime documents.

> - Total bandwidth history of these running relays, over time.

We can sum up data from bandwidth documents for this.

> - Maybe a graph showing the total number of bytes ever contributed
>   by these relays? That would impress people perhaps.

Sure, same data as above.

> - Total consensus weight of these running relays, over time.

We only have total consensus weight *fraction*, but yes.

> - Something emphasizing duration -- e.g. the total consensus weight of
>   the subset of the relays that have been in the consensus for 90% of
>   the past month, 2 months, 6 months, etc. Are there better ideas here
>   I hope? We'll want to be cognizant that if we're in the first week
>   of the challenge, the 2 month graph will be empty and thus look sad.

Not sure what the 90% part is for, but yes, graphs with total consensus
weight fraction are doable.

Regarding the sad-looking 2 month graph, we can easily define the data
when the challenge starts and not show graphs until they make sense.
Note that the current intervals for most data are 1 week, 1 month, 3
months, 1 year, and 5 years.

> - Something comparing the above numbers to the total numbers. Given how
>   huge some of the relays are lately, it would be easily to visualize
>   the new contribution as a tiny irrelevant fraction, which could be
>   disheartening to new relay operators even if their relays will actually
>   become a big deal with some patience. What are some strategies for
>   making this work right? E.g. a layer graph showing y layered on top of
>   x where y is the new contribution, rather than a percentage-of-total
>   graph that shows approximately 0%.

Absolute contributions to consensus weight are not available, just
relative fractions.

> We could also imagine more niche categories. For example, if we're hoping
> to get people to sign up relays at universities, we could imagine that
> the folks running the challenge give us a list of fingerprints of relays
> that self-identify as being at universities, and then we do up the same
> set of graphs with that subset of relays.

Sure, that's doable.

> So, Christian, others, how much of this is possible as-is or with some
> limited tweaking, with Globe and related scripts? I am hoping the answer
> is most of it. :) I also cc Karsten because a lot of this overlaps with
> the metrics scripts, but I am expecting Karsten to push back against
> the idea of integrating these measurements more with the metrics project.

Right, adding this to the metrics website is not a good idea, because
then we'd have to parse raw relay descriptors.

Somebody else to include here is Sreenatha who has done a pretty good
job processing Onionoo data for the t-shirt yes/no ticket #9889.

> Any other ideas for what to measure to help people know whether their
> contribution is being worthwhile?

Not yet, but new ideas may arise when we start working on the code.

> [*] Please don't take this mail as any official announcement, or timeline,
> or any of that. At this point we need to collect people to help make
> this happen, not collect news stories.

What's the timeline for this?  This requires some non-trivial coding
time, and I'm not sure how to prioritize this over existing things on my
todo list.

All the best,
Karsten