Re: aarch64 reliability issues

On Sun, 2025-05-18 at 02:43 +0000, John Klos wrote:
> shamshu: {12} tail /root/rolling.log
> 
> [ 8888900.8992799] /: create/symlink failed, no inodes free
> Connection to shamshu.zia.io closed by remote host.
> Connection to shamshu.zia.io closed.
> 

So the "no inodes free" is a misleading error message. No surprise
there really.

That message almost certainly means that symlink(2) returned an ENOSPC
error which has a very long description in the manual page.

I'm going to guess that the kernel encountered an undocumented error
and returned ENOSPC. IIRC POSIX constrains what errors a system call is
allowed to return and so maybe the kernel needs to get a bit creative
sometimes.

I suspect that you are running out of memory (yes, on the systems with
lots of memory). I am not a wizard when it comes to tracking down
memory usage, but I have seen amd64 systems with 32GB RAM have problems
like yours. I'm not sure which versions of NetBSD I have seen this on,
but I don't recall seeing it recently on 10.1 or 10.99.12. I might have
seen this on 10.0.

My particular problem seemed to be the ZFS memory pool taking and not
giving back when it should. I note that you do not seem to be using
ZFS. I just keep an eye on memory usage using top. I used to have to
reboot my build boxes when the "Pools" number got up towards 32GB.

Ngā mihi,
Lloyd