WO and Search Engines

Hi all,

I know there's been discussion on this before, I was just wondering  
if anyone has come up with a good solution.  This is what I am  
finding at the moment:

I have my WO app running and so far this month MSN has hit it 35,000  
times and Google around 6,000 times.  This means that each time it  
hits the site a new session is being created, due to the fact that  
the site uses sessions and that both of these search engines are  
hitting random pages, not like a normal user would browse so there is  
no consistency.  Sometimes they hit with a session ID in the URL but  
it makes no difference as those sessions would have expired so a new  
one is created.

In order to try and keep the load to a minimum, I now identify every  
hit and if it's a bot, I set the timeout on the session to be low.   
Does anyone know the minimum?  I've set it to '20' but it does not  
appear to time out for 5 mins or so.  Despite this, it still seems to  
be overloading the server.  The CPU is locking up requiring a  
restart.  I have had situation where there have been more than 2000  
sessions created in a small timeframe.  Since I have optimised it  
some more, I have got that down to a max of about 40 concurrent  
sessions if the search engines hit the site hard.  I did not think  
this should be enough to cause problems, but apparently it is as the  
server hangs and all they can tell me is that there were a lot of  
Java/WebObjects processes running.

Anyone had similar issues or found a nice solution?

Kind regards,
David Griffith.
http://www.ipodshoppers.com