Weird regression in 14.4 wrt ZFS and/or vnodes

Hi all,

we run a couple (about 80) FreeBSD hosts with jails for customer hosting environments.

Most of the jails are still on FreeBSD 14.3-latest and will be updated to 14.4 and Q3 package
repo in two weeks. We separate staging and production systems so we updated all staging
systems just yesterday and in preparation also updated and rebooted all of the hosts that
run staging systems.

Since the update from 14.3 to 14.4 some of the hosts show this behaviour:

- very high CPU utilisation up to the point applications are noticeably slowed down
- CPU is in system, not in user
- processes show state "vnode" or "vdbat" in top

Historically (we started with FreeBSD 10 or 11 with this platform) we limited the maximum
ARC size to a quarter of the RAM or rarely less if the RAM size was increased after initial
provisioning of the system.

To make a long story short - picture a system with 192 GB of physical memory and a pool built
from a single mirror vdev (SSD).

-	sysctl vfs.zfs.arc.max=34359738368 # (32 G) --> system comes to a crawl with all cores busy in system
-	sysctl vfs.zfs.arc.max=68719476736 # (64 G) --> system runs just fine like it did all the time before

Oddly setting vfs.zfs.arc.max=0 also exhibits the "CPU busy" behaviour. I would like to remove the
explicit limit entirely across all our servers, since at least in my experience current FreeBSD and ZFS
is well capable of managing ARC size vs. memory pressure from applications. But it seems not to work
as expected.

*Any* friggin' idea what might be going on? What can I do to provide more data points? I am a bit at a
loss - experienced in debugging performance problems on the application side with ktrace/dwatch
but if everything happens in system, I am at a bit of a loss.

Ah ... no excessive I/O on the SSDs.

Seems like I am not alone:
https://github.com/openzfs/zfs/issues/18397

So any help very much appreciated.

Question about a short term workaround - if setting the ARC to 32 G max slows down the system
considerably and setting it to 64 G makes it fly again - what will happen if I set it to 64 G and the
system only has 64 G memory in total? Will that behave like setting it to 0 and the system will auto
adapt or will bad things happen?

Anyway, kind regards and thanks in advance,
Patrick
-- 
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Sophienstr. 187
76185 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
[email protected]

AG Mannheim 108285
Geschäftsführer: Daniel Lienert, Fabian Stein