[tor-bugs] #16520 [Analysis]: Run some onion services to observe crawling trends

[Tor Bug Tracker & Wiki]

#16520: Run some onion services to observe crawling trends
------------------------------+---------------------
 Reporter:  arma              |          Owner:
     Type:  project           |         Status:  new
 Priority:  normal            |      Milestone:
Component:  Analysis          |        Version:
 Keywords:  SponsorR, tor-hs  |  Actual Points:
Parent ID:                    |         Points:
------------------------------+---------------------
 We know some research groups that are doing full crawling of onion
 services. We also know that Ahmia et al are doing it. I keep hearing these
 days about big security companies selling "onion intelligence" or the
 like.

 What are the characteristics of these crawls? Are many of them one level
 deep, or k levels deep, or full crawls? Do they obey robots.txt? Do they
 identify themselves by their user agent? Do they visit urls that are
 embedded in html comments that humans would never find? Do they de-
 obfuscate urls and visit those? Do they get suckered by web tarpits that
 produce infinite pages? Are the crawling trends going up quickly or
 slowly?

 We should consider running a couple of onion services with various
 characteristics, and monitor their usage and see if we learn anything.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/16520>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online