[tor-dev] Metrics Plans

[Damian Johnson]

> Here, I think it is realistic to try and use and import all the fields available from metrics-db-*.
> My PoC is overly simplistic in this regard: only relay descriptors, and only a limited subset of data fields is used in the schema, for the import.

I'm not entirely sure what fields that would include. Two options come
to mind...

* Include just the fields that we need. This would require us to
update the schema and perform another backfill whenever we need
something new. I don't consider this 'frequent backfill' requirement
to be a bad thing though - this would force us to make it extremely
easy to spin up a new instance which is a very nice attribute to have.

* Make the backend a more-or-less complete data store of descriptor
data. This would mean schema updates whenever there's a dir-spec
addition [1]. An advantage of this is that the ORM could provide us
with stem Descriptor instances [2]. For high traffic applications
though we'd probably still want to query the backend directly since we
usually won't care about most descriptor attributes.

> The idea would be import all data as DB fields (so, indexable), but it makes sense to also import raw text lines to be able to e.g. supply the frontend application with raw data if needed, as the current tools do. But I think this could be made to be a separate table, with descriptor id as primary key, which means this can be done later on if need be, would not cause a problem. I guess there's no need to this right now.

I like this idea. A couple advantages that this could provide us are...

* The importer can provide warnings when our present schema is out of
sync with stem's Descriptor attributes (ie. there has been a new
dir-spec addition).

* After making the schema update the importer could then run over this
raw data table, constructing Descriptor instances from it and
performing updates for any missing attributes.

Cheers! -Damian

[1] https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt
[2] This might be a no-go. Stem Descriptor instances are constructed
from the raw descriptor content, and needs it for str(), get_bytes(),
and signature validation. If we don't care about those we can subclass
Descriptor and overwrite those methods.