Re: Apache prefork overload caused by Googlebot crawling many host-routed sites
Frank Gingras <[email protected]> Wed, 25 Feb 2026 09:48:54 -0500
| Newsgroups | gmane.comp.apache.user |
|---|---|
| Message-ID | <CAOY50mtoEqQK1Ufu3n2wA3_4JFaC0uwkUzJS-4YkZMaoGobEHA@mail.gmail.com> |
--0000000000000d783f064ba71b98 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 25, 2026 at 5:47=E2=80=AFAM Phong Thai <[email protected]> wro= te: > Yes, I=E2=80=99m aware of fake Googlebot traffic and that is a valid conc= ern. > > I=E2=80=99m verifying Googlebot using reverse DNS (crawl-*.googlebot.com)= , > and I do see both real Googlebot and a significant amount of > cloud-provider traffic (Azure/AWS) spoofing UA. > > I already filter a large portion of obvious bot traffic at the > network / firewall level, but the overload still occurs specifically > when legitimate crawlers hit many hostnames in parallel. > > The difficulty is that with prefork, processes are spawned very early > during vhost and rewrite evaluation, so even valid crawlers can > exhaust memory before any application-level throttling applies. > > I=E2=80=99m trying to understand if there are Apache-level techniques > to reduce rewrite / vhost routing cost per request, > without blocking or misleading real Googlebot. > > On Wed, Feb 25, 2026, 4:15=E2=80=AFPM Marc <[email protected]> wrote= : > >> Are you sure it is googlebot and not fake bots. I have 400k requests per >> day from shit Microsoft Azure mostly. Try and filter out the crap, so yo= u >> have more resources left for real traffic. >> >> I have a honeypot page with robots.txt, >> 5m cron, everything >100 requests goes to ipset blacklist. >> Everthing blacklisted is redicted to lightweight html only page. >> >> I think this only works on ipv4 as these are not abundant. >> >> PS. maybe crawl delay in robots? >> PPS. upgrading also helps with performance >> >> >> > >> > >> > Hello, >> > >> > I=E2=80=99m looking for advice on handling crawler-driven overlo= ad in an >> > Apache >> > prefork environment. >> > >> > Environment: >> > - Apache httpd with prefork MPM >> > - CentOS 7.4 >> > - ~2 CPU / 4 GB RAM >> > - prefork must remain in use >> > >> > Architecture summary: >> > - Multiple main domains >> > - Tens of thousands of very small sites, each with its own >> hostname >> > - All hostnames are routed through a central VirtualHost using >> > vhost-level rewrite rules (no .htaccess) >> > - Each hostname maps dynamically to a directory such as: >> > /app/sites/{unique-sub-domain-slug}/ >> > >> > Under normal conditions the system behaves well. >> > >> > Issue: >> > When Googlebot crawls these small sites, Apache load spikes >> > severely >> > (load averages > 200). httpd processes grow rapidly and many sit= es >> > become unreachable until crawler activity subsides. Main domains >> > remain >> > responsive during these events. >> > >> > Steps already taken: >> > - All rewrite logic moved from .htaccess to VirtualHost >> > - AllowOverride disabled >> > - Conservative timeouts and connection limits applied >> > - Resources increased compared to previous smaller deployment >> > >> > This same design handled ~150 sites reasonably well in the past. >> > With a >> > much larger number of sites, overload now happens daily. >> > >> > My questions: >> > - Is this a known failure mode of prefork under heavy crawler >> > activity? >> > - Are there Apache-level techniques to limit crawler impact >> without >> > blocking Googlebot? >> > - In similar setups, what usually becomes the bottleneck first: >> > rewrite >> > processing, filesystem checks, or process spawning? >> > >> > Any insight or real-world experience would be greatly appreciate= d. >> > The solution is really to use the event mpm here - why are you bound to use the prefork approach? With prefork, the only way to scale is to pre-spawn tons of workers up to 80% of your available memory for httpd, and make sure the processes are not killed. --0000000000000d783f064ba71b98 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Feb 25,= 2026 at 5:47=E2=80=AFAM Phong Thai <<a href=3D"mailto:[email protected]= m">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_qu= ote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20= 4);padding-left:1ex"><div dir=3D"auto">Yes, I=E2=80=99m aware of fake Googl= ebot traffic and that is a valid concern.<div dir=3D"auto"><br></div><div d= ir=3D"auto">I=E2=80=99m verifying Googlebot using reverse DNS (crawl-*.<a h= ref=3D"http://googlebot.com" target=3D"_blank">googlebot.com</a>),</div><di= v dir=3D"auto">and I do see both real Googlebot and a significant amount of= </div><div dir=3D"auto">cloud-provider traffic (Azure/AWS) spoofing UA.</di= v><div dir=3D"auto"><br></div><div dir=3D"auto">I already filter a large po= rtion of obvious bot traffic at the</div><div dir=3D"auto">network / firewa= ll level, but the overload still occurs specifically</div><div dir=3D"auto"= >when legitimate crawlers hit many hostnames in parallel.</div><div dir=3D"= auto"><br></div><div dir=3D"auto">The difficulty is that with prefork, proc= esses are spawned very early</div><div dir=3D"auto">during vhost and rewrit= e evaluation, so even valid crawlers can</div><div dir=3D"auto">exhaust mem= ory before any application-level throttling applies.</div><div dir=3D"auto"= ><br></div><div dir=3D"auto">I=E2=80=99m trying to understand if there are = Apache-level techniques</div><div dir=3D"auto">to reduce rewrite / vhost ro= uting cost per request,</div><div dir=3D"auto">without blocking or misleadi= ng real Googlebot.</div></div><br><div class=3D"gmail_quote"><div dir=3D"lt= r" class=3D"gmail_attr">On Wed, Feb 25, 2026, 4:15=E2=80=AFPM Marc <<a h= ref=3D"mailto:[email protected]" target=3D"_blank">Marc@f1-outsourcing= .eu</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"marg= in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e= x">Are you sure it is googlebot and not fake bots. I have 400k requests per= day from shit Microsoft Azure mostly. Try and filter out the crap, so you = have more resources left for real traffic.<br> <br> I have a honeypot page with robots.txt,<br> 5m cron, everything >100 requests goes to ipset blacklist.<br> Everthing blacklisted is redicted to lightweight html only page.<br> <br> I think this only works on ipv4 as these are not abundant.<br> <br> PS. maybe crawl delay in robots?<br> PPS. upgrading also helps with performance<br> <br> <br> > <br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Hello,<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0I=E2=80=99m looking for advice on handling c= rawler-driven overload in an<br> > Apache<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0prefork environment.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Environment:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Apache httpd with prefork MPM<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- CentOS 7.4<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- ~2 CPU / 4 GB RAM<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- prefork must remain in use<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Architecture summary:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Multiple main domains<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Tens of thousands of very small sites, eac= h with its own hostname<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- All hostnames are routed through a central= VirtualHost using<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0vhost-level rewrite rules (no .htaccess)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Each hostname maps dynamically to a direct= ory such as:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0/app/sites/{unique-sub-domain-slug}/<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Under normal conditions the system behaves w= ell.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Issue:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0When Googlebot crawls these small sites, Apa= che load spikes<br> > severely<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(load averages > 200). httpd processes gr= ow rapidly and many sites<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0become unreachable until crawler activity su= bsides. Main domains<br> > remain<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0responsive during these events.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Steps already taken:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- All rewrite logic moved from .htaccess to = VirtualHost<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- AllowOverride disabled<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Conservative timeouts and connection limit= s applied<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Resources increased compared to previous s= maller deployment<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0This same design handled ~150 sites reasonab= ly well in the past.<br> > With a<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0much larger number of sites, overload now ha= ppens daily.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0My questions:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Is this a known failure mode of prefork un= der heavy crawler<br> > activity?<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- Are there Apache-level techniques to limit= crawler impact without<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0blocking Googlebot?<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0- In similar setups, what usually becomes th= e bottleneck first:<br> > rewrite<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0processing, filesystem checks, or process sp= awning?<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0Any insight or real-world experience would b= e greatly appreciated.<br></blockquote></div></blockquote><div><br></div><d= iv>The solution is really to use the event mpm here - why are you bound to = use the prefork approach?</div><div><br></div><div>With prefork, the only w= ay to scale is to pre-spawn tons of workers up to 80% of your available mem= ory for httpd, and make sure the processes are not killed.=C2=A0</div></div= ></div> --0000000000000d783f064ba71b98--