Re: robots.txt? off-topic
"Matthew Baker" <[email protected]> Thu, 24 Oct 2002 12:18:15 +0100
| Newsgroups | gmane.comp.web.jigsaw |
|---|---|
| Message-ID | <[email protected]> |
It is a file that allows you to specify which parts of the site are not visited by web crawlers. See http://www.robotstxt.org/ for details. Matt. ----- Original Message ----- From: "Shantz" <[email protected]> To: <[email protected]> Sent: Wednesday, October 23, 2002 5:35 PM Subject: robots.txt? off-topic > > > > > I've been using jigsaw to serve a webpage for a while. > When looking at the log, I often see what appears to be webcrawlers > doing a GET on robots.txt. I have never had such a file. Does anyone > know what this is about? > > Mike > > >