[tor-bugs] #17321 [CollecTor]: Index to better support downloaders

[Tor Bug Tracker & Wiki]

#17321: Index to better support downloaders
-----------------------------+-----------------
     Reporter:  atagar       |      Owner:
         Type:  enhancement  |     Status:  new
     Priority:  major        |  Milestone:
    Component:  CollecTor    |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |    Sponsor:
-----------------------------+-----------------

Comment (by karsten):

 Oh, we'll need the "last_modified" field, so that applications can decide
 whether they'll need to download a file or not.  We could also use a file
 digest here, but it's probably easier to keep the "last_modified" field.
 Though we should probably switch to the ISO format.

 I think I'd also want to avoid using parts of the path as field names.
 Maybe we can use one or two object types for directories and files with
 specified field names.  For example, a directory object could contain a
 "path" and two optional arrays with "directories" and "files", and a file
 object could contain a "path", a "size", and a "last_modified" time.

 And maybe we could add an optional "types" field that helps applications
 filter files based on the contained descriptor type(s).  (Some tarballs
 may contain more than one descriptor type, like bridge descriptors and
 microdescriptors.)  We could even use that "types" field for directory
 objects if a directory only contains files of the given descriptor
 type(s).

 How's this?

 {{{
 {
     "path": "https://collector.torproject.org/",
     "directories": [
         {
             "path": "archive/",
             "directories": [
                 {
                     "path": "relay-descriptors/",
                     "files": [
                         {
                             "path": "certs.tar.xz",
                             "size": 80400,
                             "last_modified": "2015-10-10 03:39",
                             "types": [
                                 "dir-key-certificate-3"
                             ]
                         }
                     ],
                     "directories": [
                         {
                             "path": "consensuses/",
                             "types": [
                                 "network-status-consensus-3"
                             ],
                             "files": [
                                 {
                                     "path": "consensuses-2007-10.tar.xz",
                                     "size": 1061648,
                                     "last_modified": "2012-05-15 14:35"
                                 },
                                 {
                                     "path": "consensuses-2007-11.tar.xz",
                                     "size": 6810308,
                                     "last_modified": "2012-05-15 14:35"
                                 }
                             ]
                         }
                     ]
                 }
             ]
         }
     ]
 }
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/17321#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online