[tor-dev] Introducing CollecTor

[Karsten Loesing]

[Replying on list with Damian's permission.]

On 06/06/14 17:21, Damian Johnson wrote:
>> I'm currently extending metrics-lib to fetch descriptors from CollecTor.
>>  Maybe Damian extends Stem to do the same (or accepts patches for that
>> purpose).
> 
> Probably obvious, but what's the advantage of this over fetching new
> descriptors from authorities and mirrors? What kind of CollecTor
> methods did you have in mind?

If you're only interested in new descriptors, then the authorities and
mirrors are the place to go.  But if you care about having the full
history of descriptors, then you should go to CollecTor.  Imagine that
your tool might break for a few hours.  There's no way to get the
missing data from the directory authorities or mirrors anymore.  For
applications like ExoneraTor, where you need to store every single
consensus, this can make the difference.  The same holds for Onionoo and
the metrics website, though it's less critical to miss a few hours.

Also, if you only care about relay descriptors, then you should be fine
asking directory authorities and mirrors.  But sanitized bridge
descriptors, exit lists, bridge pool assignments, and Torperf
measurements are only available from CollecTor.

Note that there's also a disadvantage of fetching from CollecTor: it's a
single point of failure.  Now, we could add mirrors, but it would be
even better to set up a second CollecTor instance (ideally run by
another person) in parallel to the current first instance, and have the
two exchange data.  Future work, on my list.

I guess the above should be added to the CollecTor homepage.  Do you
have any suggestions where you would have expected to read about this,
or do you have further points to add?  I'll add something next week.

All the best,
Karsten