RE: Inktomi heavily slurped
wizardsandwars <[email protected]> 12 Sep 2003 16:18:59 -0000
| Newsgroups | gmane.comp.web.oscommerce.general |
|---|---|
| Message-ID | <6aa8d857ad843ad61934687a00f7c2a5@osCommerce-Forums> |
This message was sent from: General Support http://forums.oscommerce.com/viewtopic.php?p=228450#228450 ---------------------------------------------------------------- Well, the easiest way would be to look in your whos_online section of admin when the inktomi bot is there, and visually scan the ip addresses that belong to inktomi. You should not see any SIDs for those urls. Another way would be to go to your favorite spider simulator, and use it to spider your site. Right after kicking off a spidering session from the sim, check your logs for the auser agent of that simulator, and add it to the list of user agents in your spider killer. hen spider your site again using the spider sim, and see if it picks up any SIDs. It should not.