Re: Apache prefork overload caused by Googlebot crawling many host-routed sites
Phong Thai <[email protected]> Wed, 25 Feb 2026 17:47:12 +0700
| Newsgroups | gmane.comp.apache.user |
|---|---|
| Message-ID | <CABh_vNRfoTfrzMxMBLf56=_MHYCaFX0LG_ftA=XdZ6oQ6smOrQ@mail.gmail.com> |
--000000000000bcf172064ba3ba6e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yes, I=E2=80=99m aware of fake Googlebot traffic and that is a valid concer= n. I=E2=80=99m verifying Googlebot using reverse DNS (crawl-*.googlebot.com), and I do see both real Googlebot and a significant amount of cloud-provider traffic (Azure/AWS) spoofing UA. I already filter a large portion of obvious bot traffic at the network / firewall level, but the overload still occurs specifically when legitimate crawlers hit many hostnames in parallel. The difficulty is that with prefork, processes are spawned very early during vhost and rewrite evaluation, so even valid crawlers can exhaust memory before any application-level throttling applies. I=E2=80=99m trying to understand if there are Apache-level techniques to reduce rewrite / vhost routing cost per request, without blocking or misleading real Googlebot. On Wed, Feb 25, 2026, 4:15=E2=80=AFPM Marc <[email protected]> wrote: > Are you sure it is googlebot and not fake bots. I have 400k requests per > day from shit Microsoft Azure mostly. Try and filter out the crap, so you > have more resources left for real traffic. > > I have a honeypot page with robots.txt, > 5m cron, everything >100 requests goes to ipset blacklist. > Everthing blacklisted is redicted to lightweight html only page. > > I think this only works on ipv4 as these are not abundant. > > PS. maybe crawl delay in robots? > PPS. upgrading also helps with performance > > > > > > > > Hello, > > > > I=E2=80=99m looking for advice on handling crawler-driven overloa= d in an > > Apache > > prefork environment. > > > > Environment: > > - Apache httpd with prefork MPM > > - CentOS 7.4 > > - ~2 CPU / 4 GB RAM > > - prefork must remain in use > > > > Architecture summary: > > - Multiple main domains > > - Tens of thousands of very small sites, each with its own hostna= me > > - All hostnames are routed through a central VirtualHost using > > vhost-level rewrite rules (no .htaccess) > > - Each hostname maps dynamically to a directory such as: > > /app/sites/{unique-sub-domain-slug}/ > > > > Under normal conditions the system behaves well. > > > > Issue: > > When Googlebot crawls these small sites, Apache load spikes > > severely > > (load averages > 200). httpd processes grow rapidly and many site= s > > become unreachable until crawler activity subsides. Main domains > > remain > > responsive during these events. > > > > Steps already taken: > > - All rewrite logic moved from .htaccess to VirtualHost > > - AllowOverride disabled > > - Conservative timeouts and connection limits applied > > - Resources increased compared to previous smaller deployment > > > > This same design handled ~150 sites reasonably well in the past. > > With a > > much larger number of sites, overload now happens daily. > > > > My questions: > > - Is this a known failure mode of prefork under heavy crawler > > activity? > > - Are there Apache-level techniques to limit crawler impact witho= ut > > blocking Googlebot? > > - In similar setups, what usually becomes the bottleneck first: > > rewrite > > processing, filesystem checks, or process spawning? > > > > Any insight or real-world experience would be greatly appreciated= . > --000000000000bcf172064ba3ba6e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto">Yes, I=E2=80=99m aware of fake Googlebot traffic and that= is a valid concern.<div dir=3D"auto"><br></div><div dir=3D"auto">I=E2=80= =99m verifying Googlebot using reverse DNS (crawl-*.<a href=3D"http://googl= ebot.com">googlebot.com</a>),</div><div dir=3D"auto">and I do see both real= Googlebot and a significant amount of</div><div dir=3D"auto">cloud-provide= r traffic (Azure/AWS) spoofing UA.</div><div dir=3D"auto"><br></div><div di= r=3D"auto">I already filter a large portion of obvious bot traffic at the</= div><div dir=3D"auto">network / firewall level, but the overload still occu= rs specifically</div><div dir=3D"auto">when legitimate crawlers hit many ho= stnames in parallel.</div><div dir=3D"auto"><br></div><div dir=3D"auto">The= difficulty is that with prefork, processes are spawned very early</div><di= v dir=3D"auto">during vhost and rewrite evaluation, so even valid crawlers = can</div><div dir=3D"auto">exhaust memory before any application-level thro= ttling applies.</div><div dir=3D"auto"><br></div><div dir=3D"auto">I=E2=80= =99m trying to understand if there are Apache-level techniques</div><div di= r=3D"auto">to reduce rewrite / vhost routing cost per request,</div><div di= r=3D"auto">without blocking or misleading real Googlebot.</div></div><br><d= iv class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gm= ail_attr">On Wed, Feb 25, 2026, 4:15=E2=80=AFPM Marc <<a href=3D"mailto:= [email protected]">[email protected]</a>> wrote:<br></div><blo= ckquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #c= cc solid;padding-left:1ex">Are you sure it is googlebot and not fake bots. = I have 400k requests per day from shit Microsoft Azure mostly. Try and filt= er out the crap, so you have more resources left for real traffic.<br> <br> I have a honeypot page with robots.txt,<br> 5m cron, everything >100 requests goes to ipset blacklist.<br> Everthing blacklisted is redicted to lightweight html only page.<br> <br> I think this only works on ipv4 as these are not abundant.<br> <br> PS. maybe crawl delay in robots?<br> PPS. upgrading also helps with performance<br> <br> <br> > <br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Hello,<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0I=E2=80=99m looking for advice on handling c= rawler-driven overload in an<br> > Apache<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0prefork environment.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Environment:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Apache httpd with prefork MPM<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- CentOS 7.4<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- ~2 CPU / 4 GB RAM<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- prefork must remain in use<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Architecture summary:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Multiple main domains<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Tens of thousands of very small sites, eac= h with its own hostname<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- All hostnames are routed through a central= VirtualHost using<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0vhost-level rewrite rules (no .htaccess)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Each hostname maps dynamically to a direct= ory such as:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0/app/sites/{unique-sub-domain-slug}/<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Under normal conditions the system behaves w= ell.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Issue:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0When Googlebot crawls these small sites, Apa= che load spikes<br> > severely<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(load averages > 200). httpd processes gr= ow rapidly and many sites<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0become unreachable until crawler activity su= bsides. Main domains<br> > remain<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0responsive during these events.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Steps already taken:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- All rewrite logic moved from .htaccess to = VirtualHost<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- AllowOverride disabled<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Conservative timeouts and connection limit= s applied<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Resources increased compared to previous s= maller deployment<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0This same design handled ~150 sites reasonab= ly well in the past.<br> > With a<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0much larger number of sites, overload now ha= ppens daily.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0My questions:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Is this a known failure mode of prefork un= der heavy crawler<br> > activity?<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Are there Apache-level techniques to limit= crawler impact without<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0blocking Googlebot?<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- In similar setups, what usually becomes th= e bottleneck first:<br> > rewrite<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0processing, filesystem checks, or process sp= awning?<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Any insight or real-world experience would b= e greatly appreciated.<br> </blockquote></div> --000000000000bcf172064ba3ba6e--