[metrics-bugs] #20994 [Metrics/Onionoo]: invalid first_seen timestamp on bridges

[Tor Bug Tracker & Wiki]

#20994: invalid first_seen timestamp on bridges
-----------------------------+------------------------------
 Reporter:  cypherpunks      |          Owner:  metrics-team
     Type:  defect           |         Status:  needs_review
 Priority:  Medium           |      Milestone:
Component:  Metrics/Onionoo  |        Version:
 Severity:  Normal           |     Resolution:
 Keywords:                   |  Actual Points:
Parent ID:                   |         Points:
 Reviewer:                   |        Sponsor:
-----------------------------+------------------------------
Changes (by karsten):

 * status:  new => needs_review


Comment:

 Okay, I'm more confident now that the fix suggested above is sane.  Here's
 why:

 The `updatedNodes` data structure has the sole purpose of creating a set
 of all fingerprints for `updateNodeDetailsStatuses()`, which is step 4 of
 4 in the update process.

 However, it doesn't make any sense to include a fingerprint which was
 never seen in a consensus or bridge network status and for which we don't
 have a `NodeStatus` instance.  There's nothing that the non-existent
 `NodeStatus` could contribute to the `DetailsStatus`, and there's no need
 to create a mostly empty `NodeStatus` for a node that was never listed in
 a consensus or bridge network status.  In fact, we shouldn't even return
 the node in a query, because the only proof of its existence comes from
 self-published descriptors.

 Regarding the `updatedNodes` set, we can as well go through `knownNodes`.
 In fact, at one point we're adding all `knownNodes` to `updatedNodes`
 anyway, so they contain the same entries.  We can simply kick out
 `updatedNodes` and use `knownNodes` in `updateNodeDetailsStatuses()`.

 Here's what I think happened that led to this bug: We learned about a
 bridge from a bridge server descriptor and created a mostly-empty
 `NodeStatus` for it in `updateNodeDetailsStatuses()`, including a
 `first_seen` date of 1970-01-01.  At that time we didn't see this bridge
 in any bridge network status.  But some time later we did see it in a
 status, which is how `last_seen` was updated.  With the fix we wouldn't
 have a `NodeStatus` until first seeing the bridge in a bridge network
 status.

 There, the good news is that this very likely fixes the issue.  The bad
 news is that I'll have to set up a temporary Onionoo instance and import
 '''all''' bridge network statuses in order to get correct `first_seen`
 dates for all bridges.  I'll start doing that now, but it could easily
 take days or even weeks.  Transitioning to the new, correct data will be
 fun, but there's still time to think about making that transition
 smoother.

 Please review
 [https://gitweb.torproject.org/user/karsten/onionoo.git/log/?h=task-20994
 my branch task-20994].

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20994#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online