Re: WO and Search Engines

You may consider the robots.txt file along with the Google Sitemap  
feature helpful.
https://www.google.com/webmasters/sitemaps/

When you supply this file (can be dynamically generated) it keeps  
google from crawling your website since you are telling it where all  
the pages are. I understand that it will still open sessions, etc.  
Just a thought.

  - Ken


On Mar 15, 2006, at 7:59 AM, Jerry W. Walker wrote:

> Hi, David,
>
> You exclude robots from a server by creating a file on the server  
> which specifies the access policy for robots.
>
> Make the file accessible via HTTP on the local URL "/robots.txt".
>
> See:
>
>      http://www.robotstxt.org/wc/norobots.html
>
> Here's a sample file:
> ----------------------------------------
> # robots.txt for http://www.example.com/
> User-agent: *
> Disallow: /cyberworld/map/ # This is an infinite virtual URL space
> Disallow: /tmp/ # these will soon disappear
> Disallow: /foo.html
> ----------------------------------------
>
> Regards,
> Jerry
>
> On Mar 15, 2006, at 6:45 AM, David Griffith wrote:
>
>> Hi all,
>>
>> I know there's been discussion on this before, I was just  
>> wondering if anyone has come up with a good solution.  This is  
>> what I am finding at the moment:
>>
>> I have my WO app running and so far this month MSN has hit it  
>> 35,000 times and Google around 6,000 times.  This means that each  
>> time it hits the site a new session is being created, due to the  
>> fact that the site uses sessions and that both of these search  
>> engines are hitting random pages, not like a normal user would  
>> browse so there is no consistency.  Sometimes they hit with a  
>> session ID in the URL but it makes no difference as those sessions  
>> would have expired so a new one is created.
>>
>> In order to try and keep the load to a minimum, I now identify  
>> every hit and if it's a bot, I set the timeout on the session to  
>> be low.  Does anyone know the minimum?  I've set it to '20' but it  
>> does not appear to time out for 5 mins or so.  Despite this, it  
>> still seems to be overloading the server.  The CPU is locking up  
>> requiring a restart.  I have had situation where there have been  
>> more than 2000 sessions created in a small timeframe.  Since I  
>> have optimised it some more, I have got that down to a max of  
>> about 40 concurrent sessions if the search engines hit the site  
>> hard.  I did not think this should be enough to cause problems,  
>> but apparently it is as the server hangs and all they can tell me  
>> is that there were a lot of Java/WebObjects processes running.
>>
>> Anyone had similar issues or found a nice solution?
>>
>> Kind regards,
>> David Griffith.
>
> --
> __ Jerry W. Walker,
>    WebObjects Developer/Instructor for High Performance Industrial  
> Strength Internet Enabled Systems
>
>     [email protected]
>     203 278-4085        office
>
> _______________________________________________
> WebObjects-dev mailing list
> [email protected]
> http://www.omnigroup.com/mailman/listinfo/webobjects-dev
>