[tor-bugs] #13600 [Onionoo]: Improve bulk imports of descriptor archives

[Tor Bug Tracker & Wiki]

#13600: Improve bulk imports of descriptor archives
-----------------------------+-----------------
     Reporter:  karsten      |      Owner:
         Type:  enhancement  |     Status:  new
     Priority:  normal       |  Milestone:
    Component:  Onionoo      |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+-----------------

Comment (by karsten):

 @iwakeh

 I don't have a good answer for you, because I didn't have the chance to go
 through all comments on this ticket and the others yet.  But if I were to
 re-import descriptor archives into a new Onionoo instance, I'd do the
 following:

  - Use latest master of the official repository, nothing else.
  - Decompress (but not extract) tarballs using `unxz`.
  - Start with importing a single tarball or all tarballs of a single
 month, then try with three months, then twelve, etc.  You'll probably run
 into out-of-memory problems at some point, and you'll have to find out how
 many tarballs you can process at once.  Keep in mind that tarballs got
 bigger and bigger over time.
  - Once an import run completes, move away tarballs, because otherwise
 they will be re-imported.
  - Make backups of the `status/` directory after each import run.

 Sorry that this is not as convenient as it should be.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/13600#comment:17>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online