[tor-dev] Extending Pyonionoo to provide statistics

[Karsten Loesing]

On 10/13/12 4:30 PM, Damian Johnson wrote:
> Hi Karsten. This is actually really good timing. As you probably
> noticed from tor-commits@ I just pushed support for network status
> documents (v2 and v3 votes, consensus, and microdescriptor
> consensuses). It's the combination of a month of work from Ravi and a
> couple more from me.

Neat!

> Stem's descriptor parsing functionality is one of its most well
> developed features and I'm anxious for it to start to get some users.
> The feature gap between stem and metrics-lib is now pretty small, so
> I'd love for it to start to take over some of metrics-lib's
> responsibilities (and hopefully in turn get direct involvement from
> you and Sathyanarayanan so it'll better meet your needs).

Sounds great!  I'd say let's give stem some usage then.

>> We would start with daily users, both direct and bridge users, and later add
>> aggregate statistics about relays and bridges, and after that torperf
>> performance statistics.
> 
> Gotcha. As I understand it the document types that stem is still missing are...
> 
> * microdescriptors
> * bridge pool assignments
> * exit list entry
> * torperf output
> 
> So what you'll need from me is primarily the ability to parse torperf
> output? Or is there another document type that I'm missing?

No, we wouldn't need Torperf parsing support yet.  Extending Pyonionoo
to provide Torperf statistics would be one of the last steps.

>> I think we'd need Damian's help for the descriptor-parsing side
> 
> Happy to help, though I'm not entirely sure yet what kind of
> additional parsing support you'll need.

I was mostly thinking of your help if things don't work as expected.
Also, having your input on Pyonionoo's design might be very valuable.

>> Sathya, Damian, can you already say how much of 2013 you will be around
>> for doing Tor stuff and how much time you think you could spend on this
>> project?
> 
> For my part a fair bit. This is, of course, a hobby that I do outside
> of a full time job so work and life might get in the way. That said,
> any project that includes collaborating with other developers to make
> use of stem goes to the top of my todo list.

Cool!

>> I'm thinking one ticket for implementing
>> usage statistics in Pyonionoo, a second ticket for the censorship
>> detector integration which depends on the first ticket, and two more
>> tickets for network statistics and torperf statistics integration into
>> Pyonionoo.  I'd probably suggest the first two tickets only for trying
>> to get funding from sponsor F year 3 and save the other two tickets for
>> later.
> 
> Hmm, aren't those last two tickets the only ones where I'd be
> involved? It sounds like this project mostly concerns pyonionoo so
> guess we should wait for input from Sathyanarayanan...

That's correct.

I just created #7154, #7155, and #7156 (combining the last two tickets).

Thanks,
Karsten