Re: Identifying causes of "seems busy" entries
Alex <[email protected]> Wed, 6 Aug 2025 14:38:21 -0400
| Newsgroups | gmane.comp.php.general |
|---|---|
| Message-ID | <CAB1R3sgLVvFp6H4xxXBku5CmEbMoDcBuMriS8DViTCYqXfXHaQ@mail.gmail.com> |
--000000000000dafaea063bb6a5a6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you all for your incredible feedback. I'm going to study/review it over the coming days and follow up with questions :-) On Wed, Aug 6, 2025 at 4:46=E2=80=AFAM AllenJB <[email protected]> wr= ote: > On 06/08/2025 00:15, Alex wrote: > > Hi, > > I have a fedora40 system with apache-2.24.62 and php-8.3.20 (upgrading > > to fedora42 soon). I'm trying to identify how best to optimize php-fpm > > for problems involving "seems busy" errors. I'm sure this is a FAQ, > > but I'm really stuck. I've been using php-fpm and tweaking it for > > years, but our site is constantly growing and I don't know what I'm > > missing. Apache and php-fpm are the only processes on the system. The > > server is a Xeon Gold 6128 @ 3.4Ghz with 128GB of RAM. > > > > [05-Aug-2025 15:36:58] WARNING: [pool mysite] seems busy (you may need > > to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 > > children, there are 91 idle, and 137 total children > > Ref: > > https://github.com/php/php-src/blob/1820dd9b61319b79d6a2f7570c24449c1a352= 50b/sapi/fpm/fpm/fpm_process_ctl.c#L417 > > This message is issued whenever PHP-FPM considers spawning 8 or more > children at the same time. It does not take account of the number of > idle children or the maximum number of children. (It seems to me the > conditions required to emit this message could be improved, but this is > how it's currently implemented) > > (I say "considers", because from the code below and a very brief search > of where this value is set, it looks like it will only spawn this many > if there aren't enough idle servers already - the number to consider is > based on doubling the number of processes spawned in the previous cycle, > so this message is in fact emitted when FPM spawns processes 4 cycles in > a row and hasn't reached pm.min_spare_servers, if I'm reading this > correctly) > > Based on my reading of the FPM process management code, and your > settings (from which I presume you run a relatively busy server), I > could see this message being emitted quite frequently, particularly > after FPM has just been restarted. For example: > > * FPM starts and starts 110 workers (start_servers) > * 10 workers are immediately busy (and the number of busy workers > remains at that level for the next 5 FPM process management cycles) > * cycle 1: FPM spawns 1 additional worker (wanted workers to reach > minimum is now 9) > * cycle 2: FPM spawns 2 additional workers (wanted workers to reach > minimum is now 7) > * cycle 3: FPM spawns 4 additional workers (wanted workers to reach > minimum is now 3) > * cycle 4: FPM emits the above message and considers spawning 8 > additional workers (but this gets floored to 3, the minimum required to > reach min_spare_servers) > > You'd see a similar situation in any case where the number of busy > workers increases by 8+ and stays at that level (or higher) for at least > 4 FPM process management cycles. > > (I could be wrong but I think a process management cycle occurs every 1 > second, so the above example scenario occurs over 5 seconds) > > The message is emitted each time the process management cycle runs, so > if the number of busy workers is 16+, the message will be emitted twice > in a row, 32+ will be 3 times in a row, and so on. > > The point I try to make here is that FPMs heuristic for "pool seems > busy" is fairly naive, and may trigger quite frequently on servers that > exceed a certain level of traffic (or at least changes in the amount of > traffic that is maintained for 5 seconds or more) > > > I note that you've implemented the FPM status page, but are you aware > that you can implement your own monitoring and alerting by utilizing it > (specifically the openmetrics format) with tools such as Grafana + > Prometheus? This would allow you to set up your own alerts based on your > setup, rather than relying on FPM's log warnings. > > For the available formats and how to use them see > https://www.php.net/manual/en/fpm.status.php#fpm.status.parameters > > --000000000000dafaea063bb6a5a6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Thank you all for your=C2=A0incredible feedback. I= 9;m going to study/review it over the coming days and follow up with questi= ons :-)</div><br><div class=3D"gmail_quote gmail_quote_container"><div dir= =3D"ltr" class=3D"gmail_attr">On Wed, Aug 6, 2025 at 4:46=E2=80=AFAM AllenJ= B <<a href=3D"mailto:[email protected]">[email protected]</a= >> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On = 06/08/2025 00:15, Alex wrote:<br> > Hi,<br> > I have a fedora40 system with apache-2.24.62 and php-8.3.20 (upgrading= <br> > to fedora42 soon). I'm trying to identify how best to optimize php= -fpm <br> > for problems involving "seems busy" errors. I'm sure thi= s is a FAQ, <br> > but I'm really stuck. I've been using php-fpm and tweaking it = for <br> > years, but our site is constantly growing and I don't know what I&= #39;m <br> > missing. Apache and php-fpm are the only processes on the system. The = <br> > server is a Xeon Gold 6128 @ 3.4Ghz with 128GB of RAM.<br> ><br> > [05-Aug-2025 15:36:58] WARNING: [pool mysite] seems busy (you may need= <br> > to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8= <br> > children, there are 91 idle, and 137 total children<br> <br> Ref: <br> <a href=3D"https://github.com/php/php-src/blob/1820dd9b61319b79d6a2f7570c24= 449c1a35250b/sapi/fpm/fpm/fpm_process_ctl.c#L417" rel=3D"noreferrer" target= =3D"_blank">https://github.com/php/php-src/blob/1820dd9b61319b79d6a2f7570c2= 4449c1a35250b/sapi/fpm/fpm/fpm_process_ctl.c#L417</a><br> <br> This message is issued whenever PHP-FPM considers spawning 8 or more <br> children at the same time. It does not take account of the number of <br> idle children or the maximum number of children. (It seems to me the <br> conditions required to emit this message could be improved, but this is <br= > how it's currently implemented)<br> <br> (I say "considers", because from the code below and a very brief = search <br> of where this value is set, it looks like it will only spawn this many <br> if there aren't enough idle servers already - the number to consider is= <br> based on doubling the number of processes spawned in the previous cycle, <b= r> so this message is in fact emitted when FPM spawns processes 4 cycles in <b= r> a row and hasn't reached pm.min_spare_servers, if I'm reading this = <br> correctly)<br> <br> Based on my reading of the FPM process management code, and your <br> settings (from which I presume you run a relatively busy server), I <br> could see this message being emitted quite frequently, particularly <br> after FPM has just been restarted. For example:<br> <br> * FPM starts and starts 110 workers (start_servers)<br> * 10 workers are immediately busy (and the number of busy workers <br> remains at that level for the next 5 FPM process management cycles)<br> * cycle 1: FPM spawns 1 additional worker (wanted workers to reach <br> minimum is now 9)<br> * cycle 2: FPM spawns 2 additional workers (wanted workers to reach <br> minimum is now 7)<br> * cycle 3: FPM spawns 4 additional workers (wanted workers to reach <br> minimum is now 3)<br> * cycle 4: FPM emits the above message and considers spawning 8 <br> additional workers (but this gets floored to 3, the minimum required to <br= > reach min_spare_servers)<br> <br> You'd see a similar situation in any case where the number of busy <br> workers increases by 8+ and stays at that level (or higher) for at least <b= r> 4 FPM process management cycles.<br> <br> (I could be wrong but I think a process management cycle occurs every 1 <br= > second, so the above example scenario occurs over 5 seconds)<br> <br> The message is emitted each time the process management cycle runs, so <br> if the number of busy workers is 16+, the message will be emitted twice <br= > in a row, 32+ will be 3 times in a row, and so on.<br> <br> The point I try to make here is that FPMs heuristic for "pool seems <b= r> busy" is fairly naive, and may trigger quite frequently on servers tha= t <br> exceed a certain level of traffic (or at least changes in the amount of <br= > traffic that is maintained for 5 seconds or more)<br> <br> <br> I note that you've implemented the FPM status page, but are you aware <= br> that you can implement your own monitoring and alerting by utilizing it <br= > (specifically the openmetrics format) with tools such as Grafana + <br> Prometheus? This would allow you to set up your own alerts based on your <b= r> setup, rather than relying on FPM's log warnings.<br> <br> For the available formats and how to use them see <br> <a href=3D"https://www.php.net/manual/en/fpm.status.php#fpm.status.paramete= rs" rel=3D"noreferrer" target=3D"_blank">https://www.php.net/manual/en/fpm.= status.php#fpm.status.parameters</a><br> <br> </blockquote></div></div> --000000000000dafaea063bb6a5a6--