Re: Apache prefork overload caused by Googlebot crawling many host-routed sites

It was thus said that the Great Phong Thai once stated:
> Hello,
> 
> I’m looking for advice on handling crawler-driven overload in an Apache 
> prefork environment.

 ... 

> - Are there Apache-level techniques to limit crawler impact without 
> blocking Googlebot?

  I found mod_limitipconn that works well in my case.  It limits the number
of concurrent connections from a single IP address to a configurable number. 
I changed the code to return a 429 instead of 503, but that's the only
change I've made.  I set globally the number of concurrent connections to 20
(again, from a single IP address).

  -spc