Re: WO and Search Engines

Hi, David,

You exclude robots from a server by creating a file on the server  
which specifies the access policy for robots.

Make the file accessible via HTTP on the local URL "/robots.txt".

See:

      http://www.robotstxt.org/wc/norobots.html

Here's a sample file:
----------------------------------------
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
----------------------------------------

Regards,
Jerry

On Mar 15, 2006, at 6:45 AM, David Griffith wrote:

> Hi all,
>
> I know there's been discussion on this before, I was just wondering  
> if anyone has come up with a good solution.  This is what I am  
> finding at the moment:
>
> I have my WO app running and so far this month MSN has hit it  
> 35,000 times and Google around 6,000 times.  This means that each  
> time it hits the site a new session is being created, due to the  
> fact that the site uses sessions and that both of these search  
> engines are hitting random pages, not like a normal user would  
> browse so there is no consistency.  Sometimes they hit with a  
> session ID in the URL but it makes no difference as those sessions  
> would have expired so a new one is created.
>
> In order to try and keep the load to a minimum, I now identify  
> every hit and if it's a bot, I set the timeout on the session to be  
> low.  Does anyone know the minimum?  I've set it to '20' but it  
> does not appear to time out for 5 mins or so.  Despite this, it  
> still seems to be overloading the server.  The CPU is locking up  
> requiring a restart.  I have had situation where there have been  
> more than 2000 sessions created in a small timeframe.  Since I have  
> optimised it some more, I have got that down to a max of about 40  
> concurrent sessions if the search engines hit the site hard.  I did  
> not think this should be enough to cause problems, but apparently  
> it is as the server hangs and all they can tell me is that there  
> were a lot of Java/WebObjects processes running.
>
> Anyone had similar issues or found a nice solution?
>
> Kind regards,
> David Griffith.

--
__ Jerry W. Walker,
    WebObjects Developer/Instructor for High Performance Industrial  
Strength Internet Enabled Systems

     [email protected]
     203 278-4085        office