[metrics-bugs] #29369 [Metrics/Onionperf]: Fix message logging and filtering

[Tor Bug Tracker & Wiki]

#29369: Fix message logging and filtering
---------------------------------------+--------------------------------
 Reporter:  irl                        |          Owner:  metrics-team
     Type:  defect                     |         Status:  assigned
 Priority:  Medium                     |      Milestone:
Component:  Metrics/Onionperf          |        Version:
 Severity:  Normal                     |     Resolution:
 Keywords:  metrics-team-roadmap-2020  |  Actual Points:  0.4
Parent ID:                             |         Points:  1.0
 Reviewer:                             |        Sponsor:  Sponsor59-must
---------------------------------------+--------------------------------
Changes (by karsten):

 * status:  accepted => assigned
 * owner:  karsten => metrics-team
 * points:  1 => 1.0
 * actualpoints:   => 0.4


Comment:

 After having looked at past logs and thinking about possible fixes I came
 up with a rather trivial fix: let's give up on using a strict date filter
 when analyzing logs and simply include all transfers and control events
 found in the parsed log files.

 Example: op-ab's log files from 2019-01-13 contain measurements from
 2019-01-12 and -13, because log rotation didn't happen at 2019-01-12
 23:59:59. With the current code, the analysis file produced at 2019-01-13
 23:59:59 is called `2019-01-13.onionperf.analysis.json.xz` and contains
 measurements from 2019-01-13 only. After the suggested change the file
 would still have that file name but contain all measurements from
 2019-01-12 and -13 as found in the log files.

 (A possible alternative that I had been thinking about before was that the
 analysis of a single pair of log files could produce more than one
 analysis files depending on how many dates are covered in the log files.
 This can get nasty, though. We'd have to consider all sorts of edge cases
 where two log files contain measurements for the same date. But really, we
 don't have to make sure that measurements go into analysis files with a
 specific date. Let's just not do this anymore.)

 I didn't work on the implementation yet and would like to leave this to
 others, also as a way to sanity check the concept. If I were to implement
 this I'd probably avoid using the `date_filter` parameter when doing the
 nightly analysis and introduce some sort of `date_prefix` parameter. We
 should just use that `date_filter` by default for all analysis files, and
 we might consider deprecating the `date_prefix` parameter in the user
 interface. When testing this, we'll have to ensure that it works for the
 nightly case as well as the reprocessing case.

 Returning to metrics-team.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29369#comment:8>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online