[tor-dev] Comparing Stem, metrics-lib, and zoossh

Tue Jan 12 04:38:46 UTC 2016

Hi Karsten, implemented Stem counterparts of these (see attached). On
one hand the code is delightfully simple, but on the other
measurements I got were quite a bit slower. Curious to see what you
get when running at the same place you took your measurements.

Cheers! -Damian

On Thu, Jan 7, 2016 at 8:02 AM, Karsten Loesing <karsten at torproject.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 03/01/16 21:25, Damian Johnson wrote:
>> Nice! Few questions...
>>
>> * Where are your metrics-lib scripts used for the benchmarks?
>> Should be easy for me to write stem counterparts once I know what
>> we're running. I'll later be including our demo scripts with the
>> benchmarks later so if possible comments would be nice so they're
>> good examples for newcomers to our libraries.
>
> I'm planning to clean up this code before committing it to a real
> repository, but here's the unclean version in a pastebin:
>
> http://pastebin.com/PEXD09jF
>
>> * Which exact tarballs are you parsing? It would be useful if we
>> ran all our benchmarks on the same host with the same data.
>
> I'm using tarballs from CollecTor, except for microdescriptors which
> I'm processing as described below.
>
> Agreed about running this on the same host in the future.
>
>> * Please take note somewhere of the metric-lib commit id used
>> since I'll want to include that later when we add the results to
>> our site.
>
> Good idea.
>
> For now, I think I'll wait for you to write similar benchmarks for
> Stem to learn whether I need to write any more for metrics-lib.  And
> then I'll clean up things more on my side and commit them somewhere
> more serious than pastebin.
>
>> Sorry I didn't get to this for the task exchange. Been focusing on
>> Nyx so quite a few things have fallen off my radar.
>
> Sure, no worries at all. Very much looking forward to your results!
>
> All the best,
> Karsten
>
>
>>
>> Cheers! -Damian
>>
>>
>> On Sun, Jan 3, 2016 at 9:56 AM, Karsten Loesing
>> <karsten at torproject.org> wrote: Hi Damian,
>>
>> I'm digging out this old thread, because I think it's still
>> relevant.
>>
>> I started writing some performance evaluations for metrics-lib and
>> got some early results.  All examples read a monthly tarball from
>> CollecTor and do something trivial with each contained descriptor
>> that requires parsing them.  Here are the average processing times
>> by type:
>>
>> server-descriptors-2015-11.tar.xz: 0.334261 ms
>> server-descriptors-2015-11.tar: 0.285430 ms
>> extra-infos-2015-11.tar.xz: 0.274610 ms extra-infos-2015-11.tar:
>> 0.215500 ms consensuses-2015-11.tar.xz: 255.760446 ms
>> consensuses-2015-11.tar: 246.713092 ms
>> microdescs-2015-11.tar.xz[*]: 0.099397 ms
>> microdescs-2015-11.tar[*]: 0.066566 ms
>>
>> [*] The microdescs* tarballs contain microdesc consensuses and
>> microdescriptors, but I only cared about the latter; what I did is
>> extract tarballs, delete microdesc consensuses, and re-create and
>> re-compress tarballs
>>
>> These evaluations were all run on a Core i7 with 2GHz using an SSD
>> as storage.
>>
>> Any surprises in these results so far?
>>
>> Would you want to move forward with the comparison and also
>> include Stem?  (And, Philipp, would you want to include Zoossh?)
>>
>> All the best, Karsten
>>
>>
>> On 01/10/15 09:28, Karsten Loesing wrote:
>>>>> Hello Philipp and iwakeh, hello list,
>>>>>
>>>>> Damian and I sat down yesterday at the dev meeting to talk
>>>>> about doing a comparison of the various descriptor-parsing
>>>>> libraries with respect to capabilities, run-time performance,
>>>>> memory usage, etc.
>>>>>
>>>>> We put together a list of things we'd like to compare and
>>>>> tests we'd like to run that we thought we'd want to share
>>>>> with you. Damian and I will both be working on these for
>>>>> metrics-lib for a short while and then switch to Stem.
>>>>> Please feel free to join us in these effort. The result is
>>>>> supposed to live on Stem's home page unless somebody comes up
>>>>> with a better place.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> All the best, Damian and Karsten
>>>>>
>>>>>
>>>>> On 30/09/15 10:57, Karsten Loesing wrote:
>>>>>> 1. capabilities - supported descriptor types - all the ones
>>>>>> on CollecTor's formats.html - hidden service descriptors
>>>>>> (have an agreed @type for that) - getting/producing
>>>>>> descriptors - reading from file/directory - reading from
>>>>>> tarballs - reading from CollecTor's .xz-compressed tarballs
>>>>>> - fetching from CollecTor - downloading from directories
>>>>>> (authorities or mirrors) - generating (for unit test) -
>>>>>> recognizing @type annotation - inferencing from file name -
>>>>>> keeping reading history - user documentation - validation
>>>>>> (format, crypto, successful sanitization) - packages
>>>>>> available - how much usage by (large) applications
>>>>>
>>>>>> 2. performance (CPU time, memory overhead) - compression:
>>>>>> .xz-compressed tarballs/decompressed tarballs/plain-text -
>>>>>> descriptor type: consensus, server descriptor, extra-info
>>>>>> descriptor, microdescriptors - validation: on or off
>>>>>> (allows lazy loading)
>>>>>
>>>>>> 3. tests by descriptor type - @type server-descriptor 1.0
>>>>>> - Stem's "List Outdated Relays" - average advertised
>>>>>> bandwidth - fraction of relays that can exit to port 80 -
>>>>>> @type extra-info 1.0 - sum of all written and read bytes
>>>>>> from write-history/read-history - number of countries from
>>>>>> which v3 requests were received - @type
>>>>>> network-status-consensus-3 - average number of relays with
>>>>>> Exit flag - @type network-status-vote-3 - Stem's "Votes by
>>>>>> Bandwidth Authorities" - @type dir-key-certificate-3 -
>>>>>> @type network-status-microdesc-consensus-3 1.0 - @type
>>>>>> microdescriptor 1.0 - look at single microdesc cons and
>>>>>> microdescs, compile list of extended families - fraction of
>>>>>> relays that can exit to port 80 - @type network-status-2
>>>>>> 1.0 - @type directory 1.0 - @type bridge-network-status -
>>>>>> @type bridge-server-descriptor - @type
>>>>>> bridge-server-descriptor 1.0 - @type bridge-extra-info 1.3
>>>>>> - @type bridge-pool-assignment - @type tordnsel 1.0 - @type
>>>>>> torperf 1.0
>>>>>
>>>>>> 4. action items - get in touch with Dererk for packaging
>>>>>> metrics-lib for Debian
>>>>>
>>>>>
>>>>>
>>
>>
>
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBAgAGBQJWjowbAAoJEJD5dJfVqbCr3CwH/iQK7Pj1SqhnjESa2uzGFklZ
> j1qNeS0P2VLbm3HcFZgZd7SOyY90tkYNWAcSxlMlbiLnGyn/eVKkOtlzFyqhiW6O
> y1LF/1Q5a823QFgE0x3y9NcvXmB0LUxnPMzwrWNBVuFtL/HWqF4jfnflNO9HDiCx
> GQi6GtMp+iPwOJgk6lqVpsJ28UqewymxaXHkhA99IKFNOAyMApf+F/vu+HLibQek
> P1b4fUxVPai8a/TB1bg8+Aj6EDB1PvxOBVplneV9tJsDmG8lLZCj/vb4Ko8jiCZ5
> qLndkF0WdraOR82PDeeaf6n62ca/CDxIOWQ3F6Paxa9ZmSiBdAqymBEnIwo7Gfw=
> =PbRy
> -----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: metrics.py
Type: text/x-python
Size: 3143 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20160111/dcf1531c/attachment.py>