[tor-dev] Python ExoneraTor

[Karsten Loesing]

On 09/06/14 01:26, Damian Johnson wrote:
> Oh, and another quick thought - you once mentioned that a descriptor
> search service would make ExoneraTor obsolete, and in looking it over
> I agree. The search functionality ExoneraTor provides is trivial. The
> only reason it requires such a huge database is because it's storing a
> copy of every descriptor ever made.
> 
> I suspect the actual right solution isn't to rewrite ExoneraTor at
> all, but rather develop a new service that can be queried for this
> descriptor data. That would make for a *much* more worthwhile project.
> 
> ExoneraTor? Nice to have. Descriptor archive service? Damn useful. :)

I agree, that was the idea behind Kostas' GSoC project last year.  And I
still think it's a good idea.  It's just not trivial to get right.

Regarding your comment about storing a copy of every descriptor ever
made, I believe that users trust ExoneraTor's results more if they see
the actual descriptors that lead to results.  Of course, I'm saying that
without knowing what ExoneraTor users actually want.  But let's not drop
descriptor copies from the database easily.

And, heh, when you say that the search functionality ExoneraTor provides
is trivial, a little part of me is dying.  It's the part that spent a
few weeks on getting the search functionality fast enough for
production.  That was not at all trivial.  The oraddress24, oraddress48,
and exitaddress24 fields as well as the indexes are the result of me
running lots and lots of sample queries and wondering about Postgres'
EXPLAIN ANALYZE results.  Just saying that it's not going to be trivial
to generalize the search functionality towards other fields than IP
addresses and dates.

If others want to follow, here's the SQL code I'm talking about:

https://gitweb.torproject.org/exonerator.git/blob/HEAD:/db/exonerator.sql

So, I'm happy to talk about writing a searchable descriptor archive.  It
could _start_ with ExoneraTor's functionality (minus the target address
and port thing discussed in that other email), and then we could
consider adding more searches.

Pretty sure that Kostas is reading this (in fact, I just cc'ed him), so
let me make one remark about optimizing Postgres defaults: I wrote quite
a few database queries in the past, and some of them perform horribly
(relay search) whereas others perform really well (ExoneraTor).  I
believe that the majority of performance gains can be achieved by
designing good tables, indexes, and queries.  Only as a last resort we
should consider optimizing the Postgres defaults.

You realize that a searchable descriptor archives focuses much more on
database optimization than the ExoneraTor rewrite from Java to Python
(which would leave the database untouched)?

All the best,
Karsten