Re: WO and Search Engines

On 15.03.2006 13:59 Uhr, "Jerry W. Walker" <[email protected]> wrote:

> You exclude robots from a server by creating a file on the server
> which specifies the access policy for robots.
> 
> Make the file accessible via HTTP on the local URL "/robots.txt".

A lot of search bots ignore robots.txt. Microsofts engine was one of those
in the past so a few years ago I collected IP (ranges) for these bots and
blocked them on the firewall. This brought the traffic back to normal.

This week I have switched one of our larger applications to use direct
actions for 99% of the requests. Sessions are only created if there was no
way to get around it (LongResponsePages).

So, for now, I have more db traffic, but this seems to work fine and I can
let bots in again, have bookmarkable URLs - and if I really need a session,
I ask for one. There would be an option to check the request headers before
doing so, whether the incoming request is from a search bot and return just
a place holder page without creating a session.

So, one way for you might be (I haven't tested this!) to override
"dispatchRequest" in Application, check for common search bot headers and
return a static page response if you are sure, the request comes from a
search bot.

cug