[metrics-bugs] #29374 [Metrics/Onionperf]: Analysis files sometimes present negative numbers in the payload_progress field

[Tor Bug Tracker & Wiki]

#29374: Analysis files sometimes present negative numbers in the payload_progress
field
-----------------------------------+------------------------------
 Reporter:  irl                    |          Owner:  metrics-team
     Type:  defect                 |         Status:  new
 Priority:  Medium                 |      Milestone:
Component:  Metrics/Onionperf      |        Version:
 Severity:  Normal                 |     Resolution:
 Keywords:  acute-2019-q1-planned  |  Actual Points:
Parent ID:                         |         Points:
 Reviewer:                         |        Sponsor:
-----------------------------------+------------------------------

Comment (by acute):

 Replying to [comment:3 karsten]:
 > Some thoughts on this, since I have been looking at tgen logs for a
 while now:
 >
 >  - The tgen log line you mention does not contain anything to identify
 the transfer. Would it be easier to use the next tgen log line which at
 least contains the source port? `2019-02-11 00:13:00 1549843980.555482
 [info] [shd-tgen-transport.c:203] [_tgentransport_changeState] transport
 TCP,16,localhost:127.0.0.1:43810,localhost:127.0.0.1:41204,op-
 ab.onionperf.torproject.net:137.50.19.2:80,state=CONNECT,error=NONE moving
 from state CONNECT to state INIT`; and would it still be accurate enough?

 Have had another look at it, the OP parser only looks at lines containing
 [transfer-status], [transfer-complete] and [transfer-error]. It currently
 identifies transfers by the time and transfer number only, without
 including any ports.
 The problem is that the transfer number only starts appearing in lines
 onwards after 'moving from state COMMAND to state RESPONSE', and the
 parser gets all the other information for a transfer from the tgen process
 summary report...

 Not sure where to go from here...we could change the way we identify
 reports (I agree with you that we should use both ports + time) to get
 timestamp information from the previous lines, and we add rules for
 parsing these? This would mean a major change to the OP parser.



 >  - What's the reason for overwriting this timestamp with the calculated
 value for ''complete'' transfers? This log line has usec granularity, too
 (1554084910'''.250433'''). I think it would be easier to compare partial
 progress of complete and incomplete transfers if we'd use the same
 timestamps for calculating them.
 My fear was that if the values reported by the tgen process are the
 absoulute source of truth, by changing them we'd report inaccurate
 information. I agree that calculating everything the same way is more
 consistent - we should go with this after we figure out what to do for the
 previous step.
 >  - Do we know of any tools ''relying'' on `payload_progress` numbers
 being negative for incomplete downloads? I admit that this would be a
 weird use of the data. Still, maybe worth checking if we're using them
 anywhere ourselves and probably worth writing something in the next
 release notes.

 Excellent point, will have a look at the visualization module.

 >  - You'll find more tgen logs for testing at https://op-
 ab.onionperf.torproject.net/ .
 Thanks for this!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29374#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online