[tor-bugs] #6266 [Tor]: maxmind geoip db is starting to label Tor relays as country "A1"

[Tor Bug Tracker & Wiki]

#6266: maxmind geoip db is starting to label Tor relays as country "A1"
------------------------+---------------------------------------------------
 Reporter:  arma        |          Owner:                    
     Type:  defect      |         Status:  needs_revision    
 Priority:  normal      |      Milestone:  Tor: 0.2.3.x-final
Component:  Tor         |        Version:                    
 Keywords:  tor-client  |         Parent:                    
   Points:              |   Actualpoints:                    
------------------------+---------------------------------------------------
Changes (by karsten):

  * status:  needs_review => needs_revision


Comment:

 Replying to [comment:9 nickm]:
 > I think this is a plausible approach.  It's annoyingly labor-intensive,
 though.  Here are some suggestions:
 >   * As much as possible of the above (excluding downloading the files
 and blockfinder) should be done with scripts; there should not be a
 12-step process that people need to do by hand whenever maxmind updates.

 Makes sense.  We could make deanonymind.py call blockfinder and add
 explanatory text similar to the text from my previous comment.  What
 remains is for a human to check the output of `deanonymind.py` very
 carefully.

 >   * It would be best to have some way to record the outcome of the
 manual resolution decisions, and re-apply them, so that A) it's easy to
 document which manual changes we made, and B) we don't need to resolve the
 same conflict more than once.  Could there be a file that contains the
 manual changes that we re-apply to resolve conflicts?

 Agreed.  Let's make a file for manual changes (e.g., `src/config/geoip-
 manual`) and have deanonymind.py apply it.  Here's how this file could
 look like:

 {{{
 # NL, because previous MaxMind entry 31.171.128.0-31.171.133.255 is NL,
 # and RIR delegation files say 31.171.128.0-31.171.135.255 is NL.
 # -KL 2012-11-27
 "31.171.134.0","31.171.135.255","531334656","531335167","NL","Netherlands"

 # XY, because [...]
 "[...]"
 }}}

 The script would look for an existing A1 entry with the given start and
 end address and replace it with this line.  It would also warn if it was
 unable to replace an entry.

 > > I'd need to know how we'd want to document a) the general approach
 (basically what I described in this comment) and b) the manual changes.
 >
 > The instructions for making the file should really in in a README.geoip
 file distributed with the code.  For documenting the manual changes, see
 above.

 Here's how this file could look like:

 {{{
 The IP-to-country-code file in src/config/geoip is based on MaxMind's
 GeoLite Country database, with the following modifications to entries
 mapping IP address ranges to "A1" ("Anonymous Proxy"):

  - Those "A1" entries lying inbetween two entries with the same country
    code are automatically changed to that country code.  These changes can
    be overriden by specifying a different country code for one or more of
    these entries in src/config/geoip-manual.

  - Other "A1" entries are replaced with country codes specified in
    src/config/geoip-manual, or are left as is if there is no replacement
    entry in that file.

 Run src/config/deanonymind.py for details.
 }}}

 So, the next step would be to extend `deanonymind.py` to check for
 `GeoIPCountryCSV.zip` and possibly `geoip-manual` as input and to check if
 `blockfinder` is available.  Then it would do the steps described in my
 previous commit including applying manual changes, and write its output to
 `AutomaticGeoIPCountryWhois.csv`, `ManualGeoIPCountryWhois.csv`, and
 `geoip`.  That would reduce manual steps a lot.

 I'll hack on this now and create a tor branch for it.  This might take me
 a while.  Thanks for your feedback!  If you have more ideas, please let me
 know.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/6266#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online