robots.txt links

http://www.robotstxt.org/wc/robots.html

And don't forget, the archive can help if old pages have gone offline, or to
find past revisions:
http://web.archive.org/web/*/http://info.webcrawler.com/mak/projects/robots/
http://web.archive.org/web/20010711193857/www.robotstxt.org/wc/

It would be /very/ interesting to see how various tools handled unexpected data
showing up in the robots.txt file.  Let's get a grip on the "but we'd break
legacy hacks" risk before this goes too far.

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

-Bill Kearney


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/