Huge build times increase after updating from 14.3 to 14.4
Anton Saietskii <[email protected]>
| Newsgroups | gmane.os.freebsd.stable |
|---|---|
| Message-ID | <CAA2O=b-MpKL2o4L8vPo3x+W3WT0pzrOvMXtcEmqcWQWfuqmOvQ@mail.gmail.com> |
Hi folks,
Recently, I've been upgrading from releng/14.3 to releng/14.4 and as
usual, the first thing I did after booting into new BE was 'poudriere
bulk' (also, in releng/14.4 jail). For my set of packages, full
rebuild with cold ccache takes about a day, however this time it took
_twice_ as long. I did not notice any difference in CPU frequency or
utilization during the build, everything looked completely normal.
After the build finished, compared the cold build times for ports I maintain:
==> /usr/local/poudriere/data/logs/bulk/releng143-default/2026-01-18_15h59m51s/logs/tremotesf-qt6-2.9.1.log
<==
build time: 00:09:38
==> /usr/local/poudriere/data/logs/bulk/releng144-default/2026-03-08_17h24m59s/logs/tremotesf-qt6-2.9.1.log
<==
build time: 00:23:00
==> /usr/local/poudriere/data/logs/bulk/releng135-default/2026-01-24_16h37m38s/logs/xscreensaver-6.14.log
<==
build time: 00:03:24
==> /usr/local/poudriere/data/logs/bulk/releng143-default/2026-01-24_16h41m22s/logs/xscreensaver-6.14.log
<==
build time: 00:03:20
==> /usr/local/poudriere/data/logs/bulk/releng144-default/2026-03-08_17h24m59s/logs/xscreensaver-6.14.log
<==
build time: 00:07:33
Also, unjailed 'poudriere options' for my set went up from about 16 to
22 minutes.
Quick and dirty ad-hoc benchmarks using 'openssl speed', 'lzbench',
and 'diskinfo' didn't show significant difference, just regular small
fluctuations.
I don't believe issue may be in any of the build, system, or any other
configurations, or in ccache/poudriere packages -- that didn't change.
The only things changed are: kernel, and world. Thus, I've rebooted
into 14.3 BE and observed regular 16 minutes for poudriere options,
then combined 14.3 kernel with 14.4 world and got slowdown to 22
minutes again. It seems that the problem is in the world rather than
in kernel, something like 'cc suddenly started to spend twice to
thrice more time on same code' ('poudriere options' runs 'make
config-recursive', which in turn runs 'cc' at times).
Not adding dmesg or hardware configuration yet deliberately as it
doesn't seem to matter, what matters is same hardware with the same
config became twice as slow in tasks related to building code. Oddly
enough, while maintaining performance in other tasks (for example, my
Transmission instance seeds ~300 Mbps and average CPU utilization is
~12% before and after).
Any thoughts, ideas, etc would be greatly appreciated.
--
wbr,
Anton