[metrics-team] server-descriptor not readable (2018-02-08-11-05-00-server-descriptors has >30k descriptors)

[Karsten Loesing]

Hi Kathi,

On 2018-02-09 09:41, Katharina Haselhorst wrote:
> Yes, the descriptor seems to be valid - but still the java metrics lib
> can't read it (maybe too big? or some bug in the implementation?).
> Can someone confirm this issue or is it a problem specific to my
> machine? Process runs with 100% cpu, I killed it after 10mins or so...
> Just for reference my testing code:
> 
> public static void testReadServerDescriptor(String file) {
>     DescriptorReader dr =
>     DescriptorSourceFactory.createDescriptorReader();
>     for (Descriptor d : dr.readDescriptors(new File(file))) {
>         if (!(d instanceof RelayServerDescriptor)) {
>             System.err.println("wrong descriptor type: "
>         + d.getClass());
>             continue;
>         }
>         RelayServerDescriptor descriptor = (RelayServerDescriptor) d;
>         System.out.println("parsed descriptor for relay "
>         + descriptor.getFingerprint());
>     }
> }

I just gave this a try and was able to parse the file in a bit under 20
minutes. I set -Xmx6g, though it might also work with less heap space.

So, this is a known limitation of metrics-lib which cannot handle large
descriptor files very well. We're tracking this bug here, just in case
you want to follow along:

https://trac.torproject.org/projects/tor/ticket/20395

Until then, maybe just split the file into several smaller files, right
before lines starting with "@type".

Thanks for reporting this issue!

> Regards, Kathi

All the best,
Karsten

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 528 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20180209/0aac02ce/attachment.sig>