[tor-talk] Hidden Services

[Christian Siefkes]

On 09/19/2012 10:13 AM, tor at lists.grepular.com wrote:
> On 19/09/12 06:36, grarpamp wrote:
> 
>>> People use robots.txt to indicate that they don't want their site
>>> to be added to indexes.
> 
>> They use it to indicate that they don't want their site to be
>> crawled.
> 
> In almost all cases (99% or higher), robots.txt is used to indicate
> that a site shouldn't be crawled, *because* they don't want it to be
> indexed. The intention is painfully clear...

If website owners don't want a page to be indexed, they should use the
noindex meta tag: http://en.wikipedia.org/wiki/Noindex . robots.txt is only
for crawlers that automatically follow links from one page to others.

Neither standard prevents or discourages manually setting a link to the page.

Best regards
	Christian

-- 
|------- Dr. Christian Siefkes ------- christian at siefkes.net -------
| Homepage: http://www.siefkes.net/ | Blog: http://www.keimform.de/
|    Peer Production Everywhere:       http://peerconomy.org/wiki/
|---------------------------------- OpenPGP Key ID: 0x346452D8 --
Politics is for people who have a passion for changing life but lack a
passion for living it.
        -- Tom Robbins, Even Cowgirls Get the Blues

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://lists.torproject.org/pipermail/tor-talk/attachments/20120919/2d55dc5f/attachment.pgp>