[tor-bugs] #25259 [Metrics/Statistics]: Tune advbwdist module of metrics-web

[Tor Bug Tracker & Wiki]

#25259: Tune advbwdist module of metrics-web
--------------------------------+------------------------------
 Reporter:  iwakeh              |          Owner:  metrics-team
     Type:  enhancement         |         Status:  new
 Priority:  Medium              |      Milestone:
Component:  Metrics/Statistics  |        Version:
 Severity:  Normal              |     Resolution:
 Keywords:                      |  Actual Points:
Parent ID:                      |         Points:
 Reviewer:                      |        Sponsor:
--------------------------------+------------------------------

Comment (by iwakeh):

 Please review [https://gitweb.torproject.org/user/iwakeh/metrics-
 web.git/commit/?h=task-25259 these tuning steps].
 From the commit comment:
 Processing advbwdist-validafter.csv (350M) took 150 seconds and used up to
 7G.
 Performing pre-processing separately, helping R by defining read types,
 and avoiding multiple casting operations led to halving the processing
 time (to 77 seconds) and reducing the necessary memory to about 25%
 (approx. 1.8G). The resulting advbwdist.csv are identical.

 In future it will be necessary to split the aggregating process into
 years.  Or, to store data for years that won't have changes anymore and
 combine existing with freshly computed data.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25259#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online