RE: Inktomi heavily slurped
wizardsandwars <[email protected]> 12 Sep 2003 16:52:22 -0000
| Newsgroups | gmane.comp.web.oscommerce.general |
|---|---|
| Message-ID | <9007ad2cb1c304a53ecdc418e5ead543@osCommerce-Forums> |
This message was sent from: General Support http://forums.oscommerce.com/viewtopic.php?p=228464#228464 ---------------------------------------------------------------- More info. http://forums.oscommerce.com/viewtopic.php?t=36577 http://forums.oscommerce.com/viewtopic.php?t=39566 You must be confusted, becaues Burt's spider Killer doesn't use IP addresses, it uses the user agent string. None of the URLs that Inktomi spiders should have SIDs, otherwise Inktomi will get caught in an endless loop, and will continue to spider your site all the time. I've seen this bot do over 60,000 hits in less than a month, and suck up over 2 gigs of bandwidth when it gets stuck in this loop. I'd highly recommend reading every thread you can find regarding search engines and SIDs.