[tor-bugs] #7359 [Tor]: Design/implement method for collecting/reporting statistics

[Tor Bug Tracker & Wiki]

#7359: Design/implement method for collecting/reporting statistics
------------------------------------------------------------------------+---
 Reporter:  robgjansen                                                  |          Owner:                    
     Type:  enhancement                                                 |         Status:  needs_review      
 Priority:  normal                                                      |      Milestone:  Tor: 0.2.5.x-final
Component:  Tor                                                         |        Version:                    
 Keywords:  performance, simulation, statistics, tor-relay, tor-client  |         Parent:  #7357             
   Points:                                                              |   Actualpoints:                    
------------------------------------------------------------------------+---

Comment(by karsten):

 Replying to [comment:14 nickm]:
 > Replying to [comment:13 karsten]:
 > > Replying to [comment:9 nickm]:
 > > >  * Prefer TOR_SIMPLEQ to manual list management for new queues.
 > > >  * This data structure ... how does it work out in practice? It
 looks like it's going to be hideously inefficient.
 > >
 > > I can't really think of a more efficient data structure.  What do you
 have in mind?  (I'll hold off changing the code to TOR_SIMPLEQ until I'm
 sure we want to use a queue there.)
 >
 > Hm. Assuming a 64-bit architecture, this is going to eat an 8 byte
 pointer for the "next" element, plus 8 bytes for chunk element overhead,
 for every 5-byte entry we need to hold.  We could do better by using some
 kind of dynamic array structure, which wouldn't have a per-element
 overhead...

 I'm currently running somewhat bigger Shadow simulations to compare the
 overhead, but that's going to take a while.

 > but before we go and do that, we should know whether that actually
 matters.  Hence my asking about how it actually works out in practice. If
 the memory consumption on a busy relay isn't much comparatively, then
 there's not much need to re-do this.

 I wonder, if insertion_command_queue_t is an inefficient data structure,
 wouldn't that also apply to insertion_time_queue_t?  Should we make both
 of them more efficient?  If so, can you explain your dynamic array
 structure idea in some more detail?  Maybe I can give that a try next
 week.

 > > >  * This code needs unit tests and functional abstraction badly.
 > >
 > > I think functional abstraction is improved and code is now easier to
 test.  I didn't write tests yet, because I'm not quite sure where to
 start.  Would you mind writing one example test that I could use as a
 start for these functions?
 > >  - connection_buckets_note_empty_ts in connection.c
 > >  - bucket_millis_empty in connection.c
 > >  - sum_up_cell_stats_by_command in control.c
 > >  - append_cell_stats_by_command in control.c
 > >  - format_cell_stats in control.c
 >
 > Well, format_cell_stats is pretty easy: you would make one or more
 circuit_t* object with the cell statistics you want set on it, and then
 pass them to format_cell_stats, and then check whether the output is as
 you expect and whether the circuit is munged.  You could stick this in an
 existing test_*.c module in src/tests/ , or add a new module as
 appropriate.  For information on writing actual tinytest tests, see
 src/ext/tinytest*.
 >
 > Is there more you need to know there?  I'm not clear on what aspect of
 writing unit tests isn't making sense.

 The part where these tests should go was a bit unclear to me, but now it's
 clearer, I think.  Please see my updated morestats4 branch for a first
 attempt.

 > > Are there any other new functions that need testing badly?
 >
 > Our default should not be "let's test the functions that need it badly".
 Our default should be "test whatever new and changed functions we can."

 Right, I meant the latter. :)

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/7359#comment:15>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online