Re: ZFS deadlocks/memory accounting issues

On Mon, Mar 16, 2026 at 3:57 PM Konstantin Belousov <[email protected]> wrote:
>
> On Mon, Mar 16, 2026 at 03:08:44PM -0600, Alan Somers wrote:
> > I once saw a similar bug.  In my case I had a process that mmap()ed
> > some very large files on fusefs, consuming lots of inactive pages.
> > And when the system comes under memory pressure, it asks ARC to evict
> > first.  So the ARC would end up shrinking down to arc_min every time.
> > In my case, the solution was to set vfs.fusefs.data_cache_mode=0 .  I
> > suspect that similar bugs could be possible with UFS or tmpfs, if they
> > have giant files that are mmaped().
>
> What are 'similar bugs with UFS or tmpfs'?
> Can you please be more specific, what is the erronous behavior?

I experienced this bug in 2021, and reproduced it on both FreeBSD 12.2
and 13.0.  The setup was:

* A ZFS-root server with hundreds of GB of RAM and hundreds of TB of
ZFS, with a complicated ZFS workload.
* A custom fusefs file system.  Each fusefs mountpoint presented a
small number of files, some huge, and was backed by a file on ZFS
itself.
* A ctld target for each fusefs mountpoint, backed by one file on that
mountpoint.

"vmstat -o" showed that each of those ctld targets consume a huge
amount of inactive memory.  Basically, ctld was mmaping the whole file
and never releasing any pages.  The dtrace sdt:zfs:none:arc-needfree
probe showed that the page daemon was frequently asking ZFS to free
memory from ARC.  ZFS complied, and the ARC size would slowly shrink
down to vfs.zfs.arc_min .  In my case, there was no crash, and the OOM
killer wasn't involved, but performance suffered.  Setting
vfs.fusefs.data_cache_mode=0  was a perfect workaround for us, so I
never investigated further.

When I say that I suspect similar bugs may exist with UFS or tmpfs,
I'm suspecting that if ctld exports huge files from those file systems
on a mixed UFS/ZFS system, then they might consume huge amounts of
inactive pages.  But I've never checked.

-Alan