[tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and CPU time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and CPU time
-------------------------------+--------------------------------
 Reporter:  karsten            |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_revision
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+--------------------------------
Changes (by karsten):

 * status:  accepted => needs_revision


Comment:

 Commits a5f3d6a and 1873f12 look fine.

 But these are all just tweaks that save some memory here and there. Should
 we also think about taking a different approach that scales better by
 design? Even if we can limit memory usage to 6G, that's far too much.
 Ideally, we'd keep the `-Xmx2g` setting for all of CollecTor, or maybe
 `-Xmx4g`. But imagine what we'd have to set when bulk-importing logs in
 2019 or 2022.

 How about we sanitize logs in two steps: in the first step we scan all
 input files just for contained dates, and in the second step we iterate
 over input files in an order that lets us just keep a sliding window of
 log lines that we need to write output files.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:6>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online