Re: Apache prefork overload caused by Googlebot crawling many host-routed sites

--0000000000005e588d064b9e5d42
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 24, 2026 at 11:12=E2=80=AFPM Phong Thai <[email protected]> wr=
ote:

> Hello,
>
> I=E2=80=99m looking for advice on handling crawler-driven overload in an =
Apache
> prefork environment.
>
> Environment:
> - Apache httpd with prefork MPM
> - CentOS 7.4
> - ~2 CPU / 4 GB RAM
> - prefork must remain in use
>
> Architecture summary:
> - Multiple main domains
> - Tens of thousands of very small sites, each with its own hostname
> - All hostnames are routed through a central VirtualHost using
> vhost-level rewrite rules (no .htaccess)
> - Each hostname maps dynamically to a directory such as:
> /app/sites/{unique-sub-domain-slug}/
>
> Under normal conditions the system behaves well.
>
> Issue:
> When Googlebot crawls these small sites, Apache load spikes severely
> (load averages > 200). httpd processes grow rapidly and many sites
> become unreachable until crawler activity subsides. Main domains remain
> responsive during these events.
>
> Steps already taken:
> - All rewrite logic moved from .htaccess to VirtualHost
> - AllowOverride disabled
> - Conservative timeouts and connection limits applied
> - Resources increased compared to previous smaller deployment
>
> This same design handled ~150 sites reasonably well in the past. With a
> much larger number of sites, overload now happens daily.
>
> My questions:
> - Is this a known failure mode of prefork under heavy crawler activity?
> - Are there Apache-level techniques to limit crawler impact without
> blocking Googlebot?
> - In similar setups, what usually becomes the bottleneck first: rewrite
> processing, filesystem checks, or process spawning?
>
> Any insight or real-world experience would be greatly appreciated.
>
> Thank you.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
Why is prefork required here?  What non-thread-safe module are you loading?

4GB of RAM with large-ish processes is a recipe for disaster, either way.

--0000000000005e588d064b9e5d42
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g=
mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Feb 24,=
 2026 at 11:12=E2=80=AFPM Phong Thai &lt;<a href=3D"mailto:[email protected]=
om">[email protected]</a>&gt; wrote:<br></div><blockquote class=3D"gmail_q=
uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2=
04);padding-left:1ex">Hello,<br>
<br>
I=E2=80=99m looking for advice on handling crawler-driven overload in an Ap=
ache <br>
prefork environment.<br>
<br>
Environment:<br>
- Apache httpd with prefork MPM<br>
- CentOS 7.4<br>
- ~2 CPU / 4 GB RAM<br>
- prefork must remain in use<br>
<br>
Architecture summary:<br>
- Multiple main domains<br>
- Tens of thousands of very small sites, each with its own hostname<br>
- All hostnames are routed through a central VirtualHost using <br>
vhost-level rewrite rules (no .htaccess)<br>
- Each hostname maps dynamically to a directory such as: <br>
/app/sites/{unique-sub-domain-slug}/<br>
<br>
Under normal conditions the system behaves well.<br>
<br>
Issue:<br>
When Googlebot crawls these small sites, Apache load spikes severely <br>
(load averages &gt; 200). httpd processes grow rapidly and many sites <br>
become unreachable until crawler activity subsides. Main domains remain <br=
>
responsive during these events.<br>
<br>
Steps already taken:<br>
- All rewrite logic moved from .htaccess to VirtualHost<br>
- AllowOverride disabled<br>
- Conservative timeouts and connection limits applied<br>
- Resources increased compared to previous smaller deployment<br>
<br>
This same design handled ~150 sites reasonably well in the past. With a <br=
>
much larger number of sites, overload now happens daily.<br>
<br>
My questions:<br>
- Is this a known failure mode of prefork under heavy crawler activity?<br>
- Are there Apache-level techniques to limit crawler impact without <br>
blocking Googlebot?<br>
- In similar setups, what usually becomes the bottleneck first: rewrite <br=
>
processing, filesystem checks, or process spawning?<br>
<br>
Any insight or real-world experience would be greatly appreciated.<br>
<br>
Thank you.<br>
<br>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href=3D"mailto:[email protected]=
g" target=3D"_blank">[email protected]</a><br>
For additional commands, e-mail: <a href=3D"mailto:[email protected].=
org" target=3D"_blank">[email protected]</a><br>
<br></blockquote><div><br></div><div>Why is prefork required here?=C2=A0 Wh=
at non-thread-safe module are you loading?</div><div><br></div><div>4GB of =
RAM with large-ish processes is a recipe for disaster, either way.=C2=A0</d=
iv></div></div>

--0000000000005e588d064b9e5d42--