[tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

[Tor Bug Tracker & Wiki]

#25100: Make CollecTor's webstats module use less RAM and wall time
-------------------------------+------------------------------
 Reporter:  karsten            |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_review
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------

Comment (by iwakeh):

 I didn't find any more bugs processing the following benchmarks for
 determining how to approach the import using the two latest branches of
 this and the child ticket.

 The second quarter 2017 of logs from weschniakowii amounts to 32M
 (compressed) and can be processed in 36min using 8G.  The entire year
 won't work with just 8G.

 85min and 16G are needed for the entire available archives of meronense
 and weschniakowii together (compressed 59M).  The used heap median usage
 is 8.5G and the max 15.8G.

 So, depending on the hardware a conservative import strategy might be to
 import slices of quarters with subsequent CollecTor runs.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:18>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online