[tor-bugs] #26585 [Metrics/Onionoo]: improve AS number and name coverage (switch maxmind to RIPE Stat)

[Tor Bug Tracker & Wiki]

#26585: improve AS number and name coverage (switch maxmind to RIPE Stat)
---------------------------------+--------------------------
     Reporter:  nusenu           |      Owner:  metrics-team
         Type:  enhancement      |     Status:  new
     Priority:  Medium           |  Milestone:
    Component:  Metrics/Onionoo  |    Version:
     Severity:  Normal           |   Keywords:
Actual Points:                   |  Parent ID:
       Points:                   |   Reviewer:
      Sponsor:                   |
---------------------------------+--------------------------
 Onionoo currently uses maxmind for IP to as_number and as_name resolution.
 This is fast as it is a local DB lookup but it is less up-to-date and has
 less coverage than RIPEstat https://stat.ripe.net/

 This is a problem for tools that depend on onionoo's as_name and as_number
 fields like Relay Search, OrNetStats and OrNetRadar
 (I don't know maybe there are also others that are affected?)

 Currently it might takes weeks or months before new ASes get added to
 maxmind so this information
 is also missing when people lookup relays on Relay Search.

 As of today onionoo is missing AS level data for about 100 relays,
 but this value depends on how far we are away from the last maxmind
 update.

 How about we use RIPEstat API as a data source + local cache.

 To minimize the amount of required online queries against the RIPEstat API
 we can do the following
 to create the IP to AS map initially (pseudocode):

 if ip_prefix in cache
         use cached entry
 else
         perform an online lookup (query RIPEstat API)
         add new prefix entry to cache

 expire cache entries after 15 days?
 (it makes sense to log how many entries changed
 after 15 days so we know whether this value is to large or to small)

 This will significantly reduce the amount of required online API calls.

 To give you an idea of the scale (based on onionoo data from a random
 day in May 2018)

 total relay records: 8116
 unique IPv4 addresses: 7794
 unique IPv4 BGP prefixes: 3884

 each day about 50 new relays appear,
 lets assume the worst case (every new relay is not in an

 So on the estimated daily amount of queries you do would be around
 4000/15+50 = ~320 requests/day = 1 req every ~4 minutes
 which appears acceptable.

 IP to prefixes (this can return multiple matches), as_number and as_name
 lookup:
 https://stat.ripe.net/data/related-
 prefixes/data.json?resource=103.114.160.21


 IP to prefix and as number (no as_name) lookup:
 https://stat.ripe.net/data/network-info/data.json?resource=103.114.160.21

 ASN to as_name lookup:
 https://stat.ripe.net/data/as-overview/data.json?resource=AS40676


 documentation:
 https://stat.ripe.net/docs/data_api

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/26585>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online