[metrics-bugs] #32747 [Metrics/CollecTor]: Avoid reprocessing webstats files

[Tor Bug Tracker & Wiki]

#32747: Avoid reprocessing webstats files
-------------------------------+------------------------------
 Reporter:  karsten            |          Owner:  karsten
     Type:  defect             |         Status:  needs_review
 Priority:  Medium             |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------
Changes (by karsten):

 * status:  assigned => needs_review


Comment:

 I spent pretty much the whole week on this ticket to understand what
 exactly the current code is doing, writing tests for it, figuring out why
 tests broke due to using a security manager (!), and finally coding a
 patch that does what this ticket is about.

 I obviously didn't run this code on the server yet, but I expect that
 runtime of this module will shrink from 45 minutes every six hours to
 something around 10 minutes. And maybe we're lucky and it resolves #31901
 for us, though I didn't look into that yet.

 Please review:
  - [https://gitweb.torproject.org/user/karsten/metrics-
 base.git/commit/?h=task-32747&id=264e498f54a20f7d299daaf2533d043f880e6a8b
 metrics-base commit 264e498] which resolves a  rather unexpected issue
 with executing tests,
  -
 [https://gitweb.torproject.org/user/karsten/collector.git/commit/?h=task-32747&id=d5ca95a2bb74410004f5c4c93270f3fd90475068
 CollecTor commit d5ca95a] which adds some real tests for the webstats
 module, which I found useful before making substantial changes to code I
 didn't write myself, and
  -
 [https://gitweb.torproject.org/user/karsten/collector.git/commit/?h=task-32747&id=d7117f8c8ee946748eea4d2f2741195d4dbfe056
 CollecTor commit d7117f8] which resolves the actual issue of avoiding to
 reprocess webstats files.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/32747#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online