[tor-dev] globe compass integration

[Karsten Loesing]

Hi Christian, Sathya,

On 05/03/14 01:39, Sathyanarayanan Gunasekaran wrote:
> Hey Christian,
> 
> On Tue, Mar 4, 2014 at 2:28 PM, Christian <me at rndm.de> wrote:
>> Hi,
>> in the last couple of days I thought about a topic for a gsoc application.
>>
>> After getting ideas from Karsten and Sathya, I started making a rough
>> roadmap of things that I want to do for globe, in the next couple of
>> months (and maybe use it as a idea for a gsoc topic).
>>
>> - add new onionoo features to globe
> 
> Why? Do we want to deprecate onionoo?

Christian and I discussed this in private mail before moving this thread
to tor-dev at .  This bullet point should have said "Add support for new
Onionoo features to Globe", or something like that.  Examples are vote
details (#9778), bridge user graphs (#10331), monthly stats per
relay/bridge (#11041), list of transports (#11052), etc.  These are all
fine things to add to Globe, and maybe it's fine to mention them in the
second half of a GSoC proposal.  But they all depend on the Onionoo guy
to write code first, so a Globe GSoC project shouldn't depend on them.

>> - add compass functionality + open compass tickets to globe
>> - improve/add help documentation
> 
> These would be awesome!

Agreed!

>> The central point would be to add compass functionality to globe.
>> This has the advantage that the user can use the same application to
>> search for relay/bridge details and do queries with grouped results.
> 
> Sounds good.

Agreed!

>> This could be accomplished by changing the UI and modify advanced search
>> filters, add options for grouping and grouped search results.
>> Another suggestion was to add a group detail page that displays
>> aggregated data for multiple details.

Right.  For example, a group detail page could show bandwidth and
weights graphs for all relays in the group.

>> Currently compass pulls all onionoo relay details and queries them locally.
>> Karsten suggested that globe could use the same method and load the
>> onionoo data every hour or so and do the currently supported search and
>> compass functionality on it.
>> The idea is that globe could store the dump in a db (e.g. mongodb) and
>> use its features for searching, grouping and aggregating data.
> 
> Is this going to be part of globe-node? I don't think there is a need
> to rewrite the Compass backend(the logic) -- it exposes a REST API
> which can just be consumed by Globe. Majority of the open Compass
> tickets are bugs in the frontend.

This sounds like a fine idea to integrate Compass into Globe very
quickly and ask for user feedback early in the process.  Agile!

But in the long term I suggest we move the grouping logic of Compass to
Globe and decomission Compass.  Reasons:
 - that's one tool less to maintain, and Christian seems to be more of a
JavaScript person, not Python (IIRC);
 - Globe is going to be client-server soon in order to remove the need
for JavaScript in the browser, and then it's only a small step to do the
same things that Compass does right now;
 - Onionoo's search functionality is kinda hard to extend, so it would
be better if Globe did its own search based on locally cached documents.

However, as I said above, fine to start by querying Compass!

>> This would be a major rewrite away from calling the Onionoo-API, to
>> querying the locally available dump.
> 
> I don't think this is a good idea. There's no need to rewrite the
> existing Globe code.

(See above.)

>> I would like to know, if you think that integrating Compass into globe
>> would be a good idea?
> 
> Yes.
> 
>> Do you think that the compass integration would be a reasonable project
>> for the gsoc?
> 
> Yes. Bonus points if you can leverage the data provided by Compass for
> some nice graphs like heat maps (based on probability) or something
> like that.

I think that's something for the metrics website, not for Globe.
Onionoo and Globe are meant to provide status information on relays or
bridges, whereas the metrics website provides aggregate data covering
the entire Tor network.  There are exceptions like the bubble graphs
(https://metrics.torproject.org/bubbles.html) which use Onionoo's data,
but in theory those should be implemented in the metrics website only.
So, I'd say let's not mix scopes here, and let's not try to do too much
in a single summer.

>> Do you have other ideas on how to improve globe?
> 
> 1) Add the Compass frontend to Globe
> 2) Refactor the Compass UI -- I don't think the current compass UI is
> intuitive enough and could be extended to display more information in
> a nicer way.
> 3) Add visualizations and graphs based on Compass data

Looks like these ideas are already discussed above, unless I'm
overlooking something?

All the best,
Karsten