Re: Identifying causes of "seems busy" entries

On 06/08/2025 00:15, Alex wrote:
> Hi,
> I have a fedora40 system with apache-2.24.62 and php-8.3.20 (upgrading 
> to fedora42 soon). I'm trying to identify how best to optimize php-fpm 
> for problems involving "seems busy" errors. I'm sure this is a FAQ, 
> but I'm really stuck. I've been using php-fpm and tweaking it for 
> years, but our site is constantly growing and I don't know what I'm 
> missing. Apache and php-fpm are the only processes on the system. The 
> server is a Xeon Gold 6128 @ 3.4Ghz with 128GB of RAM.
>
> [05-Aug-2025 15:36:58] WARNING: [pool mysite] seems busy (you may need 
> to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 
> children, there are 91 idle, and 137 total children

Ref: 
https://github.com/php/php-src/blob/1820dd9b61319b79d6a2f7570c24449c1a35250b/sapi/fpm/fpm/fpm_process_ctl.c#L417

This message is issued whenever PHP-FPM considers spawning 8 or more 
children at the same time. It does not take account of the number of 
idle children or the maximum number of children. (It seems to me the 
conditions required to emit this message could be improved, but this is 
how it's currently implemented)

(I say "considers", because from the code below and a very brief search 
of where this value is set, it looks like it will only spawn this many 
if there aren't enough idle servers already - the number to consider is 
based on doubling the number of processes spawned in the previous cycle, 
so this message is in fact emitted when FPM spawns processes 4 cycles in 
a row and hasn't reached pm.min_spare_servers, if I'm reading this 
correctly)

Based on my reading of the FPM process management code, and your 
settings (from which I presume you run a relatively busy server), I 
could see this message being emitted quite frequently, particularly 
after FPM has just been restarted. For example:

* FPM starts and starts 110 workers (start_servers)
* 10 workers are immediately busy (and the number of busy workers 
remains at that level for the next 5 FPM process management cycles)
* cycle 1: FPM spawns 1 additional worker (wanted workers to reach 
minimum is now 9)
* cycle 2: FPM spawns 2 additional workers (wanted workers to reach 
minimum is now 7)
* cycle 3: FPM spawns 4 additional workers (wanted workers to reach 
minimum is now 3)
* cycle 4: FPM emits the above message and considers spawning 8 
additional workers (but this gets floored to 3, the minimum required to 
reach min_spare_servers)

You'd see a similar situation in any case where the number of busy 
workers increases by 8+ and stays at that level (or higher) for at least 
4 FPM process management cycles.

(I could be wrong but I think a process management cycle occurs every 1 
second, so the above example scenario occurs over 5 seconds)

The message is emitted each time the process management cycle runs, so 
if the number of busy workers is 16+, the message will be emitted twice 
in a row, 32+ will be 3 times in a row, and so on.

The point I try to make here is that FPMs heuristic for "pool seems 
busy" is fairly naive, and may trigger quite frequently on servers that 
exceed a certain level of traffic (or at least changes in the amount of 
traffic that is maintained for 5 seconds or more)


I note that you've implemented the FPM status page, but are you aware 
that you can implement your own monitoring and alerting by utilizing it 
(specifically the openmetrics format) with tools such as Grafana + 
Prometheus? This would allow you to set up your own alerts based on your 
setup, rather than relying on FPM's log warnings.

For the available formats and how to use them see 
https://www.php.net/manual/en/fpm.status.php#fpm.status.parameters