[tor-bugs] #31071 [Metrics/ExoneraTor]: Add a notice if we're missing data for a lookup

[Tor Bug Tracker & Wiki]

#31071: Add a notice if we're missing data for a lookup
--------------------------------+------------------------------
 Reporter:  karsten             |          Owner:  karsten
     Type:  enhancement         |         Status:  needs_review
 Priority:  Medium              |      Milestone:
Component:  Metrics/ExoneraTor  |        Version:
 Severity:  Normal              |     Resolution:
 Keywords:                      |  Actual Points:
Parent ID:                      |         Points:
 Reviewer:  irl                 |        Sponsor:
--------------------------------+------------------------------

Comment (by karsten):

 Replying to [comment:3 irl]:
 > Understanding how often this has happened would be a good start, maybe
 we don't need to have the importer do this if it's not happening that
 often, we can just detect it and manually add rows to the table.

 Here are the gaps of 4 hours or more that I found in existing data:

 {{{
 Gap of  19 hours between 2011-09-10T00:05 and 2011-09-10T19:28:46.
 Gap of  10 hours between 2011-09-10T22:30:22 and 2011-09-11T09:27:05.
 Gap of  23 hours between 2011-12-21T01:21:19 and 2011-12-22T00:23:36.
 Gap of   4 hours between 2012-01-10T03:16:24 and 2012-01-10T07:20:03.
 Gap of 111 hours between 2012-02-07T02:33:16 and 2012-02-11T18:07:47.
 Gap of  12 hours between 2012-11-09T05:06:32 and 2012-11-09T17:44:08.
 Gap of   6 hours between 2013-03-07T06:14:23 and 2013-03-07T13:03:51.
 Gap of  26 hours between 2013-03-07T15:16:50 and 2013-03-08T17:17:55.
 Gap of 156 hours between 2013-03-14T09:29:25 and 2013-03-20T22:03:41.
 Gap of   6 hours between 2013-08-08T20:07:23 and 2013-08-09T02:57:34.
 Gap of   7 hours between 2013-09-29T01:01:11 and 2013-09-29T08:04:56.
 Gap of  12 hours between 2013-10-05T14:13:06 and 2013-10-06T03:11:26.
 Gap of  11 hours between 2013-11-03T15:06:38 and 2013-11-04T02:31:19.
 Gap of   4 hours between 2013-12-24T08:33:57 and 2013-12-24T13:04:42.
 Gap of   7 hours between 2014-01-21T10:38:42 and 2014-01-21T18:14:20.
 Gap of  19 hours between 2015-10-09T14:13:37 and 2015-10-10T09:57:26.
 Gap of   8 hours between 2016-09-18T03:16:31 and 2016-09-18T12:11:46.
 Gap of  14 hours between 2017-11-19T17:12:04 and 2017-11-20T07:54:34.
 Gap of   9 hours between 2018-01-21T22:12:34 and 2018-01-22T08:10:15.
 Gap of   6 hours between 2018-01-26T16:11:36 and 2018-01-26T23:02:35.
 Gap of   5 hours between 2018-01-27T04:21:42 and 2018-01-27T09:28:47.
 Gap of   5 hours between 2018-01-27T14:53:18 and 2018-01-27T20:24:21.
 Gap of  18 hours between 2018-02-02T18:27:54 and 2018-02-03T13:06:09.
 Gap of   9 hours between 2018-02-25T00:16:21 and 2018-02-25T10:12:04.
 Gap of   5 hours between 2018-03-03T15:54:17 and 2018-03-03T21:09:07.
 Gap of   5 hours between 2018-09-24T15:11:07 and 2018-09-24T21:09:27.
 Gap of   9 hours between 2018-12-30T23:22:06 and 2018-12-31T08:57:55.
 Gap of  13 hours between 2019-01-12T18:30:24 and 2019-01-13T08:18:33.
 Gap of 122 hours between 2019-04-25T13:13:19 and 2019-04-30T15:40:21.
 Gap of  21 hours between 2019-05-25T19:04:43 and 2019-05-26T16:09:23.
 Gap of   9 hours between 2019-06-21T21:14:21 and 2019-06-22T07:06:06.
 }}}

 Note that a 4 hour downtime wouldn't be an issue for ExoneraTor. It
 considers a previously scanned exit IP address valid for 24 hours. We
 would probably be looking for gaps 18 hours or longer.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/31071#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online