Inktomi heavily slurped
Tomcat <[email protected]> 12 Sep 2003 16:04:15 -0000
| Newsgroups | gmane.comp.web.oscommerce.general |
|---|---|
| Message-ID | <62b04dac36692bbb6decafee086ffe7f@osCommerce-Forums> |
This message was sent from: General Support http://forums.oscommerce.com/viewtopic.php?p=228440#228440 ---------------------------------------------------------------- Hi all, I get "slurped" by Inktomi seveal times a day, about 1600 hits in two months... Does anybody know how to get rid of inktomi spider ( and most of all if I SHOULD get rid of it ) ? May be with a robots.txt file ? Is it really usefull to get spidered so many times by inktomi ? Thanks Franco