Re: robots.txt? off-topic

Thanks to all for explaining this to me.
mike

----- Original Message ----- 
From: "Mudry Julien" <[email protected]>
To: "'Shantz'" <[email protected]>
Cc: "Jigsaw List (E-mail)" <[email protected]>
Sent: Thursday, October 24, 2002 2:10 AM
Subject: RE: robots.txt? off-topic


> Hello
> 
> The robots.txt file allows a webmaster to exclude some
> pages or directories from browsing by webcrawlers. It's
> a standard called "Standard for Robot Exclusion". You
> can get more information regarding it here: 
> http://www.robotstxt.org/
> 
> Specifically, to answer your question:
> http://www.robotstxt.org/wc/faq.html#log
> 
> Regards,
> 
> Julien
> 
> > -----Original Message-----
> > From: Shantz [mailto:[email protected]]
> > Sent: Thursday, October 24, 2002 10:57 AM
> > To: [email protected]
> > Subject: robots.txt? off-topic
> > 
> > 
> > 
> > 
> > 
> > 
> > I've been using jigsaw to serve a webpage for a while.
> > When looking at the log, I often see what appears to be webcrawlers
> > doing a GET on robots.txt.  I have never had such a file.  Does anyone
> > know what this is about?
> > 
> > Mike
> > 
> >