[tor-bugs] #29624 [Metrics/ExoneraTor]: New version of exit list format

[Tor Bug Tracker & Wiki]

#29624: New version of exit list format
-------------------------------------------------+-------------------------
 Reporter:  irl                                  |          Owner:  karsten
     Type:  task                                 |         Status:
                                                 |  accepted
 Priority:  Medium                               |      Milestone:
Component:  Metrics/ExoneraTor                   |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  metrics-exit-list-project metrics-   |  Actual Points:
  roadmap-2019-q2                                |
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Changes (by karsten):

 * status:  new => accepted
 * cc: metrics-team (added)
 * owner:  metrics-team => karsten


Comment:

 I already started working on this as discussed yesterday. Some comments:

 Replying to [ticket:29624 irl]:
 > We need to extend the exit list format to include:
 >
 > Metadata:
 >
 > * Source identifier
 > * Source software name
 > * Source software version
 > * Source IP address

 I already have these.

 > * Source ASN
 > * Source country

 I'm not sure about these. We would basically include these by having the
 source look up its IP address in a database. But then the result depends
 on which database (version) the source uses. Of course, whoever uses this
 information could as well look up the source IP address in the database
 (version) of their choice and discard these two fields. Maybe this means
 we shouldn't put too much effort in the source's ability to include these
 two fields. Or we could just omit them from the spec. Not sure!

 > * Source operator contact details
 >
 > Measurement results:
 >
 > * IPv6 addresses

 Yep, these make sense.

 > * Error code
 >
 > Error codes:
 >
 > * Success
 > * Unknown Failure
 > * Timeout
 > * (others we can think of now, but we can extend this later when we
 actually implement it)

 This one is tricky. I don't think that the current scanner includes scans
 that ended with unknown failures or timeouts. It includes, for each found
 exit IP address, the latest scan time of a successful run resulting in
 that IP address. It probably omits IP addresses after a given number of
 hours, but we'd have to look at the code in order to know.

 > I think we should probably have one line per measurement, so IPv4 and
 IPv6 results would be listed separately, not on the same line. In the
 future we may have differing transports to consider (TCP/QUIC/something
 else) so maybe we should not just have IPv4 vs IPv6 but some numeric
 identifier that is later extensible.

 Agreed on the IPv4/IPv6 distinction. I was thinking to simply include a
 new `ExitAddress6` line for IPv6 addresses and continue using
 `ExitAddress` for IPv4 addresses. And I'd probably simply add another
 keyword for the next transport or address version. What else do you have
 in mind?

 Relatedly, I'd want to include `OrAddress` and `OrAddress6` for the
 addresses found in the consensus. Background is that I'd like to use exit
 lists as single input document type for ExoneraTor in the future.

 > Exit lists are not currently included in torspec but probably should be.
 The specification should cover the existing format, and then also the new
 format. We should expect that we will later extend the new format with a
 signature. Maybe we should just figure that out now also.

 Turns out that specifying the existing format is not trivial. Right now
 I'm looking at metrics-lib only, but I think I'll have to look at other
 code that produces/consumes these lists. For example, it would be great to
 know whether `Published` and `LastStatus` in the current format are
 considered required or optional fields, because it would be very
 convenient to lose them in version 2. What other code I should be looking
 at?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29624#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online