robots.txt? off-topic
"Shantz" <[email protected]> Wed, 23 Oct 2002 12:35:33 -0400 (EDT)
| Newsgroups | gmane.comp.web.jigsaw |
|---|---|
| Message-ID | <018b01c27ab2$c10828e0$0501a8c0@Lindow> |
I've been using jigsaw to serve a webpage for a while. When looking at the log, I often see what appears to be webcrawlers doing a GET on robots.txt. I have never had such a file. Does anyone know what this is about? Mike