[tor-bugs] #18798 [Metrics/CollecTor]: analysis of descriptor completeness

[Tor Bug Tracker & Wiki]

#18798: analysis of descriptor completeness
-------------------------------+-----------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  task               |         Status:  needs_information
 Priority:  Medium             |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  ctip               |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+-----------------------------------

Comment (by iwakeh):

 I added the referencing descriptor analysis to the todo list.

 Another question regarding the logging statements:

 What do the following numbers mean, i.e. how reliable are these numbers?
 Also, the difference between "importing" and "downloading"?

 {{{
 2016-04-21 20:08:29 INFO org.torproject.ernie.db.relaydescs.ArchiveWriter
 checkMissingDescriptors Finished writing relay descriptors to disk.
 While importing relay descriptors from local Tor data directories, we
 stored 1 consensus(es), 0 microdesc consensus(es), 9 vote(s), 9
 certificate(s), 784 server descriptor(s), 767 extra-info descriptor(s),
 and 0 microdescriptor(s) to disk.
 While downloading relay descriptors from the directory authorities, we
 stored 0 consensus(es), 1 microdesc consensus(es), 0 vote(s), 0
 certificate(s), 44 server descriptor(s), 44 extra-info descriptor(s), and
 75 microdescriptor(s) to disk.
 Statistics on the completeness of written relay descriptors:
 V, 2016-04-21 20:00:00, 8114/8114 S (100.0%), 8111/8111 E (100.0%)
 V, 2016-04-21 20:00:00, 7934/7934 S (100.0%), 7931/7931 E (100.0%)
 V, 2016-04-21 20:00:00, 7969/7969 S (100.0%), 7966/7966 E (100.0%)
 V, 2016-04-21 20:00:00, 7934/7934 S (100.0%), 7931/7931 E (100.0%)
 V, 2016-04-21 20:00:00, 8099/8099 S (100.0%), 8096/8096 E (100.0%)
 V, 2016-04-21 20:00:00, 7934/7934 S (100.0%), 7931/7931 E (100.0%)
 V, 2016-04-21 20:00:00, 8118/8118 S (100.0%), 8115/8115 E (100.0%)
 V, 2016-04-21 20:00:00, 7934/7934 S (100.0%), 7931/7931 E (100.0%)
 V, 2016-04-21 20:00:00, 8079/8079 S (100.0%), 8076/8076 E (100.0%)
 C, 2016-04-21 20:00:00, 9/9 V (100.0%), 7155/7155 S (100.0%), 7153/7153 E
 (100.0%)
 M, 2016-04-21 20:00:00, 7155/7155 M (100.0%)
 }}}

 And:
 Why are there so many "404 Not found" like
 {{{
 2016-04-22 09:08:51 FINE
 org.torproject.ernie.db.relaydescs.RelayDescriptorDownloader
 downloadResourceFromAuthority Downloaded
 http://154.35.32.5/tor/extra/d/eba9d8540ab196082eb5b8ce5b88c264ac4682e2+ec1f657931f383050e0676c10694ac5129cd6a0f+edd2c259c793fe800cec14ffffd4ce3726f6c9ab+ee5838444f89659d0358fdee3b326f7d6fc7607a+ee8d7295f465e015d16488a043208da30f90e8c7+efc3ceacd5a50dcc8e7c1cd89c3620e75d2e03ca+efc4ed691d99f7b4d8b2a0f14418b298649ca4ac+f0512e9c7f2444e3e480a06ca1a4137e7344e97b+f076e485dd2bd10a342d0fa7d5bdbe4c4dcfd62e+f095e676bd2264b70d26bc3f4bd5d2c770ed9ca3+f0c374adf1096adc0cd54dd6d48d9fe75892d837+f0c87b38a3603cfda11f9e796da65208463a91e3+f120142e2cd82effb3f8a14a61fa62277d45a92e+f16d6a5c1d236b7a56cdf491163d825d03663068+f193d3422168b9bf1097e01cd299b2ad493e7ef6+f29c619aae973ae7052e30fc4065252ec063ca67+f2e1e462e74ba9af9a2f078906447965c3264b12+f32d0c792511b9f9d2d26b576206fc69c8388449+f4623ec336ed83c5dc7fe6a7c88d519abe345c0d+f49cdcd06bffaa28e2d7735b088a1e7af5823a65+f4e60697bc1eb4899993bea2af1593d8410c1b1e+f547156e1eec23858548818eaf94fc8495a2c355+f567b500d0a45cbb21efa8211179d53cd089af5f+f5848db6200401e1634c956
 940e69dd165211b7e+f6d11790ba60c07c6d6855725f8f93d085e5651b+f791b8a476db1e6ff0e80196bca5f5a0ee14b7d8+f805ffac5b20c0c74e654a14334ccee21b273fec+f8be01a62ce55196a5396638929f2417a56871b2+f937342ed4ddb8eaae2f24b65e060c0a1efd067e+f97802f6f800997d4c8e1d6a8810e152a1047f12+f9c67c9973a7cb17d84ed44b7fa12d0a50780273+f9d821c071760f365a515db487f8e080db295683+f9d852d3fcc1eb7f2b52ae8d3da1d5788dfbb844+f9eca45a51938dc424b84a3a0de6c05531534290+f9f68effe2b846caf01776bdb28069b28f129d24+fa33a938a577148fa6f95007ae868d890c4cc726+fa40d36e1a6c4679b5d52fb935e7da45088e7bd6+fa4d44ebe08afd4ac77bd605c795040d5e659dc7+fad968587da998d29f144990c75051d469ce7423+faf4aedbfb47ed82ae7f9dd19c53aef7e96fd037+fb48f07c350a1f8062862c89ae9a2d22ce194309+fc5ac1718bbfd61f364382f21aff31004a1476c3+fcc1184ce33e2d9262bae11c850cd61c8b054c90+fccf823342c858b0aef14d7678170e7a3db53cda+fce65f424aeda39e0fb747a8fb71c50a8f11bee9+fcf5e07b6f710e0ae5640165a77c02ccbed5c9b8+fd080aae6fc3a0c9a3672c04cafcdae4cc261a07+fd21d67244376f85f5d023253dd975c7fc19
 264a+fda2f8d27a90fbbede4b4c9cdafd1aecd10a0f63+fddc63bd930b3671c28b0478c1a0e5345db3a039+fe0f93b940bcad4c1302213b792d9d6a3d904c84+fe1a42f2db7bd684459f5cbc72a1377b04a4059a+fea1405d7d35d823625d91c25d94daa89ae28e43+fec8d0b3179d3e09081e245a9216c15147aa5717+ff04fa1e0de9f01e15193798934f027b01988a87
 -> 404 (0 bytes)
 }}}

 Does this mean that just one of the requested descriptors wasn't found?
 Or, that there was no answer, or ...?
 Hope I didn't overlook anything obvious in the tor-spec.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18798#comment:8>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online