RE: robots.txt? off-topic

Mudry Julien <[email protected]> Thu, 24 Oct 2002 05:11:33 -0400 (EDT)

Newsgroups	gmane.comp.web.jigsaw
Message-ID	<[email protected]>




Hello

The robots.txt file allows a webmaster to exclude some
pages or directories from browsing by webcrawlers. It's
a standard called "Standard for Robot Exclusion". You
can get more information regarding it here: 
	http://www.robotstxt.org/

Specifically, to answer your question:
	http://www.robotstxt.org/wc/faq.html#log

Regards,

	Julien

> -----Original Message-----
> From: Shantz [mailto:[email protected]]
> Sent: Thursday, October 24, 2002 10:57 AM
> To: [email protected]
> Subject: robots.txt? off-topic
> 
> 
> 
> 
> 
> 
> I've been using jigsaw to serve a webpage for a while.
> When looking at the log, I often see what appears to be webcrawlers
> doing a GET on robots.txt.  I have never had such a file.  Does anyone
> know what this is about?
> 
> Mike
> 
>