[tor-bugs] #20335 [Metrics/CollecTor]: ReferenceChecker causes OOM

[Tor Bug Tracker & Wiki]

#20335: ReferenceChecker causes OOM
-------------------------------+---------------------
 Reporter:  iwakeh             |          Owner:
     Type:  defect             |         Status:  new
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------

Comment (by karsten):

 Yes, metrics-lib is to blame.  Here's a small class I wrote to show that:

 {{{
 package oom;

 import org.torproject.descriptor.DescriptorReader;
 import org.torproject.descriptor.DescriptorSourceFactory;

 import java.io.File;

 public class Main {
   public static void main(String[] args) {
     DescriptorReader descriptorReader =
         DescriptorSourceFactory.createDescriptorReader();
     descriptorReader.addDirectory(new File("in"));
     descriptorReader.readDescriptors();
   }
 }
 }}}

 And here's the output, once with the default heap size and once with 8G:

 {{{
 $ javac -d bin/ -cp lib/descriptor-1.4.0.jar src/oom/Main.java && java -cp
 bin/:lib/descriptor-1.4.0.jar:lib/slf4j-api-1.7.7.jar:lib/logback-
 classic-1.1.2.jar:lib/logback-core-1.1.2.jar:lib/commons-compress-1.9.jar
 oom.Main
 19:53:31.906 [Thread-0] ERROR o.t.d.impl.DescriptorReaderImpl - Bug:
 uncaught exception or error while reading descriptors: Java heap space
 java.lang.OutOfMemoryError: Java heap space
         at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_102]
         at
 java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
 ~[na:1.8.0_102]
         at
 java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
 ~[na:1.8.0_102]
         at
 java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
 ~[na:1.8.0_102]
         at
 org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.readFile(DescriptorReaderImpl.java:378)
 ~[descriptor-1.4.0.jar:1.4.0-adf4a67]
         at
 org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.readDescriptors(DescriptorReaderImpl.java:284)
 ~[descriptor-1.4.0.jar:1.4.0-adf4a67]
         at
 org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.run(DescriptorReaderImpl.java:188)
 ~[descriptor-1.4.0.jar:1.4.0-adf4a67]
         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
 $ javac -d bin/ -cp lib/descriptor-1.4.0.jar src/oom/Main.java && java
 -Xmx8g -cp bin/:lib/descriptor-1.4.0.jar:lib/slf4j-api-1.7.7.jar:lib
 /logback-classic-1.1.2.jar:lib/logback-core-1.1.2.jar:lib/commons-
 compress-1.9.jar oom.Main
 19:54:02.620 [Thread-0] ERROR o.t.d.impl.DescriptorReaderImpl - Bug:
 uncaught exception or error while reading descriptors: null
 java.lang.OutOfMemoryError: null
         at
 java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
 ~[na:1.8.0_102]
         at
 java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
 ~[na:1.8.0_102]
         at
 java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
 ~[na:1.8.0_102]
         at
 java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
 ~[na:1.8.0_102]
         at
 org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.readFile(DescriptorReaderImpl.java:378)
 ~[descriptor-1.4.0.jar:1.4.0-adf4a67]
         at
 org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.readDescriptors(DescriptorReaderImpl.java:284)
 ~[descriptor-1.4.0.jar:1.4.0-adf4a67]
         at
 org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.run(DescriptorReaderImpl.java:188)
 ~[descriptor-1.4.0.jar:1.4.0-adf4a67]
         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
 }}}

 I can't think of a clean way to avoid this problem other than keeping
 votes in the `recent/` folder in separate files.  Seriously, I didn't
 think of this error case at all.  Wow.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20335#comment:12>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online