[tor-dev] Comparing Stem, metrics-lib, and zoossh

[Damian Johnson]

Oh, forgot to talk about compression. You can run the stem script
against compressed tarballs but python didn't add lzma support until
python 3.3...

https://stem.torproject.org/faq.html#how-do-i-read-tar-xz-descriptor-archives

I suppose we could run over bz2 or gz tarballs, or upgrade python. But
can't say the compressed benchmark is overly important.

Cheers! -Damian


On Thu, Jan 14, 2016 at 2:23 AM, Karsten Loesing <karsten at torproject.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 13/01/16 20:42, Damian Johnson wrote:
>>> This was Stem commit c01a9cda4e7699c7f4bd642c8e81ed45aab7a29b
>>> and Python version 2.7.10.
>>
>> Great, thanks! Also what was the metrics-lib and zoossh commits?
>
> metrics-lib: 8767f3e3bb8f6c9aa8cdb4c9fb0e9f2b545a7501
> java version "1.7.0_51"
>
> Zoossh: 2380e557e35532fd25870c8fc7a84a3fc951dbfc
> go version go1.5.2 darwin/amd64
>
>>> Or should we add these performance tests for metrics-lib, Stem,
>>> and Zoossh to their own repository that also comes with scripts
>>> to fetch data from CollecTor?  (Not sure if this is a smart thing
>>> to do, but I figured I should ask before adding things to the
>>> metrics-lib repository.)
>>
>> Sinister plan #572 is that I'll add these results and scripts to
>> the page we have with the library comparison. Probably tomorrow. If
>> you'd care to clean up the metrics-lib examples that would be
>> great. Otherwise I'll include what we have and you can send me
>> patches later with what you'd like.
>
> Sounds like a good plan.  Thanks for doing this!
>
> Here's one thing I realized when composing this message.  When I
> pasted Zoossh results earlier, I compared them to the results for
> metrics-lib and Stem processing tarballs.  But Zoossh can only process
> extracted tarballs.  I just re-ran metrics-lib and Stem with extracted
> tarballs and included all results below:
>
> server-descriptors-2015-11.tar.xz:
>  - metrics-lib: 0.334261 ms
>
> server-descriptors-2015-11.tar:
>  - metrics-lib: 0.285430 ms
>  - Stem: 1.02 ms (357%)
>
> server-descriptors-2015-11/:
>  - metrics-lib: 0.682293 ms
>  - Zoossh: 0.458566 ms (67%)
>  - Stem: 1.11 ms (163%)
>
> extra-infos-2015-11.tar.xz:
>  - metrics-lib: 0.274610 ms
>
> extra-infos-2015-11.tar:
>  - metrics-lib: 0.215500 ms
>  - Stem: 0.68 ms (316%)
>
> consensuses-2015-11.tar.xz:
>  - metrics-lib: 255.760446 ms
>
> consensuses-2015-11.tar:
>  - metrics-lib: 246.713092 ms
>  - Stem: 1393.10 ms (565%)
>
> consensuses-2015-11/:
>  - metrics-lib: 283.910864 ms
>  - Stem: 1303.53 ms
>  - Zoossh: 83 ms
>
> microdescs-2015-11.tar.xz[*]:
>  - metrics-lib: 0.099397 ms
>
> microdescs-2015-11.tar[*]:
>  - metrics-lib: 0.066566 ms
>  - Stem: 0.66 ms (991%)
>
> [*] The microdescs* tarballs contain microdesc consensuses and
> microdescriptors, but I only cared about the latter; what I did is
> extract tarballs, delete microdesc consensuses, and re-create and
> re-compress tarballs
>
> I'm attaching a slightly updated version of the metrics-lib code.
> It's not cleaned up, but it's what I used to perform the measurements
> above.
>
> All the best,
> Karsten
>
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQEcBAEBAgAGBQJWl3cmAAoJEJD5dJfVqbCrew8H/1JnOCbZAXD9dBQUIYsZPu//
> jngO1Khf2lcggcHcxNMkhZS67KMmdKnyu0ZdjBHGdRNLZbQhhbDf5kQVtdx4xGim
> h4+cMgNEWqVp2gvfr/pvF6luns1rXfKREN/uxs9B1zrk5nEqWjAuPWqdgFbqA3Ui
> 03VOd1inUNZ7fWFwJpvSzb5RuZHusmLAWsDXO/607cJ/Of99QUIU5NWEBwwVTvgl
> ATd1+Slo2KIsNSpVPgtIbv345X7kTs2Jvt/ZvJsotQuRzn18d0A2ZayCddpQ896Z
> gffIp+xY+19tr1x27bwbDS44Jhb3p1Y1cKqvAUjZXzk8iSecxL+kSn5JCEdl2YY=
> =7nr/
> -----END PGP SIGNATURE-----