Re: ZFS deadlocks/memory accounting issues

On Mon, Mar 16, 2026 at 3:23 PM Garrett Wollman <[email protected]> wrote:
>
> <<On Mon, 16 Mar 2026 15:08:44 -0600, Alan Somers <[email protected]> said:
>
> > A less effective workaround was to set vfs.zfs.arc.min to some
> > reasonable value.  That can prevent ARC from shrinking too far.  You
> > could try that.
>
> So far as I can tell, the ARC doesn't actually shrink, and shouldn't
> need to given the gigabytes of free physmem at the time (well,
> immediately prior).  Within 5 minutes of the crash, the total ARC size
> was 70 GiB, c_max was 127 GiB, and c_min was 4 GiB -- in practice it's
> never anywhere near that small.  The first observation after the
> server came back up, ARC size was already over 20 GiB.
>
> Either *something* is causing the kernel to think it has no free
> memory when there's actually lots, or else something is causing the
> kernel to allocate gigabytes of RAM much faster than we can observe it
> happening.
>
> There's epsilon memory in the inactive queue on this system, before or
> after the crash: it's so small I can't even see the line on the graph.
> The 24-hour maximum is 268 MiB, or about 0.2% of RAM.
>
> > Another thing you could try is to run "vmstat -o" when the system is
> > in the problematic state.
>
> What's the equivalent in DDB?  No getty, no login.
>
> -GAWollman

I don't know how to do it from ddb.  But if you dump a core file, then
you can run vmstat after rebooting like this: "vmstat -o -M
/var/crash/vmcore.XXX" .  Also, did the kernel panic, or did you
manually enter ddb?  If it paniced, can you please share the stack
trace?