[tor-commits] [metrics-web/master] Add a robots.txt to discourage indexing database lookups.

[karsten at torproject.org]

commit 46264713a550421c358e471a1dd1c50f20a46e5f
Author: Karsten Loesing <karsten.loesing at gmx.net>
Date:   Mon Sep 12 08:21:42 2011 +0200

    Add a robots.txt to discourage indexing database lookups.
---
 web/robots.txt |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/web/robots.txt b/web/robots.txt
new file mode 100644
index 0000000..3b8e76e
--- /dev/null
+++ b/web/robots.txt
@@ -0,0 +1,12 @@
+User-agent: *
+Disallow: /networkstatus.html
+Disallow: /routerdetail.html
+Disallow: /relay.html
+Disallow: /csv/
+Disallow: /serverdesc
+Disallow: /extrainfodesc
+Disallow: /descriptor.html
+Disallow: /consensus
+Disallow: /votes
+Disallow: /consensus-health.html
+