Re: nfs server issues

On Tue, Jul 7, 2026 at 5:30 AM Paul Barker <[email protected]> wrote:
>
> On Tue, 2026-07-07 at 14:29 +0300, Konstantin Belousov wrote:
> > On Tue, Jul 07, 2026 at 12:18:21PM +0100, Paul Barker wrote:
> > > On Tue, 2026-07-07 at 09:52 +0100, Richard Purdie wrote:
> > > > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> > > > > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> > > > > <[email protected]> wrote:
> > > > > >
> > > > > > nfsstat -E -s
> > > > > > Server Info:
> > > > > >       Getattr      Setattr       Lookup     Readlink         Read        Write
> > > > > >   17354100116   1674996998   5136769057        40544  18429670397    986443058
> > > > > >        Create       Remove       Rename         Link      Symlink        Mkdir
> > > > > >             0     98646613    119865835     17071853            0            0
> > > > > >         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
> > > > > >             0   1655544711            0   4129535844            0            0
> > > > > >        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
> > > > > >             0            0      8723436            0            0            0
> > > > > One more thing to note. The # of Commits is much smaller than the # of Writes
> > > > > on the top line. This "hints" that a lot of the Writes are being done File_sync.
> > > > > (To check that, you'd need to capture packets and look at them in wireshark.)
> > > > >
> > > > > If a lot of the Writes are File_sync (which means the server must commit the
> > > > > data/metadata changes to stable storage before replying), doing the "mirrored
> > > > > pair of storage devices dedicated to the ZIL" could be what you need to get
> > > > > writes to work well. (As I mentioned, the "cheat" alternative is to
> > > > > set "sync=disabled",
> > > > > but that runs a risk of data loss when the NFS server crashes/reboots.)
> > > >
> > > > I was able to find a way to increase the Open/Lock counts on the
> > > > server. Most of the locking we do on the clients is from a tool called
> > > > Bitbake and the locking is isolated to specific function. I maintain
> > > > Bitbake and wrote/maintain the locking code over the <too many> years.
> > > > I copied and pasted the relevant code straight from bitbake into a test
> > > > script:
> > > >
> > > > https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py
> > > >
> > > > (code is from
> > > > https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)
> > > >
> > > > Obviously it could easily be simplified more but I wanted to test our
> > > > actual code. The lock file name/path is hardcoded at the end of the
> > > > script but easily changed.
> > > >
> > > > I ran that script on three of the NFS clients for around 5 minutes. On
> > > > the server, the counts went from:
> > > >
> > > > nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> > > > CB                    2      6307        55        55         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> > > > CB                    3      6543        77        77         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> > > > CB                    2      6571        83        83         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> > > >
> > > > to:
> > > >
> > > > nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> > > > CB                    2      6462        65        65         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> > > > CB                    3      6679        83        83         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> > > > CB                    2      6691        88        88         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> > > >
> > > > I've then stopped the scripts and waited and the counts just stayed
> > > > there.
> > > >
> > > > I'd be very interested if someone else could reproduce that and if so,
> > > > explain what is happening and if it is an issue?
> > > >
> > > > I can't prove the write hangs are related to the counts but it does
> > > > seem there is some issue there regardless as they don't make sense.
> > >
> > > Hi all,
> > >
> > > This is a long one...
> > >
> > > I've looked into this issue with the Yocto Project autobuilder cluster
> > > alongside Richard. I've used Claude to help me navigate the Linux and
> > > FreeBSD source trees, along with manually checking some of the key
> > > findings, so this is in the territory of a "working theory" as to what's
> > > going on. Some input from folks more familiar with the FreeBSD NFS
> > > server code would definitely be helpful to confirm I'm on the right
> > > track.
> > >
> > > The issue we're seeing is probably triggered by the following sequence
> > > of events:
> > >
> > > 1) Client A opens a file on the FreeBSD server via an NFS mount. Even if
> > >    the client immediately closes the file, Linux's NFS client code will
> > >    keep the file handle cached for a short while before sending a CLOSE
> > >    to the server.
> > >
> > > 2) During this window, client B deletes or replaces the file. This sends
> > >    a REMOVE or RENAME message to the server. The FreeBSD NFS server code
> > >    unlinks the underlying file immediately.
> > >
> > > 3) Client A now releases the file, either explicitly or due to cache
> > >    timing out. At this point it sends a PUTFH/CLOSE compound message to
> > >    the NFS server, as CLOSE acts on the current file handle the PUTFH is
> > >    needed to select which file is being closed. However, the file handle
> > >    is now stale due to the removal from another client so the PUTFH
> > >    operation fails with ESTALE. The CLOSE operation is never reached, so
> > >    no resource cleanup occurs. The file handle has now leaked and will
> > >    not be released unless the client session which owns it terminates.
> > >
> > > I've confirmed that the file becomes stale with a simple experiment:
> > >
> > > 1) Open a file via a Python shell on client A. Leave this Python shell
> > >    running.
> > >
> > >     >>> fh = open("/srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt", "r")
> > >
> > > 2) Remove the file from client B:
> > >
> > >     $ rm /srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt
> > >
> > > 3) Now attempt to read the file via the open handle on client A:
> > >
> > >     >>> fh.read()
> > >     Traceback (most recent call last):
> > >       File "<python-input-2>", line 1, in <module>
> > >         fh.read()
> > >         ~~~~~~~^^
> > >     OSError: [Errno 116] Stale file handle
> > >
> > > I've also looked at the NFS server code myself on the freebsd-src
> > > stable/14 branch. In nfsrvd_compound(), case NFSV4OP_PUTFH calls
> > > nfsd_fhtovp(), which calls nfsvno_fhtovp() to resolve the file handle.
> > > This sets nd->nd_repstat to ESTALE if the file has been unlinked. This
> > > causes a break from the main loop in nfsrvd_compound() to return the
> > > error to the client, so the CLOSE operation is never processed. And if
> > > CLOSE is never processed, the open file handle leaks.
> > >
> > > This causes us two kinds of problems:
> > >
> > > 1) The hash table of open file handles grows. With >400k file handles in
> > >    the hash table we see the CPU load over 5000%. There is one file
> > >    handle table in the NFS server, with one lock, and all stateful
> > >    operations seem to need to take this lock to lookup a file handle, so
> > >    this leads to thrashing.
> > >
> > >    Linux may interpret timeouts waiting for a server response as
> > >    EREMOTEIO.
> > >
> > >    This can be somewhat ameliorated by increasing vfs.nfsd.fhhashsize to
> > >    spread the open handles across more buckets in the hash table, but
> > >    that just delays the load issues, it doesn't resolve them.
> > >
> > > 2) When the total number of open file handles + open locks hits 500,000,
> > >    we hit the default v4statelimit. This causes the server to reply to
> > >    operations with NFSERR_RESOURCE which Linux also interprets as
> > >    EREMOTEIO.
> > >
> > >    The ~483,000 open file handles in Richard's original message is
> > >    consistent with the 500k limit being reached before a few clients
> > >    were disconnected (zeroing out their open counts and reducing the
> > >    overall total).
> > >
> > >    We could increase vfs.nfsd.v4statelimit, but again this is just
> > >    delaying the inevitable issues. If we're leaking ~500k open file
> > >    handles in less than a week then no limit will last very long.
> > >
> > > So, essentially we would need to avoid the leak in the first place.
> > > Should the error handling for PUTFH on a stale file handle free the
> > > associated state? Or should a REMOVE/RENAME operation leave the file
> > > handle intact until all consumers have released the file handle (this is
> > > what a Linux NFS server seems to do)?
> > >
> > > Thank you for bearing with that long explanation! If I've misunderstood
> > > anything then please let me know and I'll take another look.
> >
> > Does the FreeBSD NFS server still reference the vnode for the deleted file
> > in your situation?  If yes, then c041b82c59e885d106f45454302bad3d9dcc58a7
> > might already help.
>
> Hi Konstantin, thanks for the info. I sadly don't think this patch will
> help, we're using ZFS as the backing store for these NFS shares and that
> commit only changes UFS/FFS code. I also don't think the NFS server
> holds a vnode reference for each open stateid but I may be wrong, I'm
> not really familiar with the code base.
You are correct. The Open/Lock structures hold file handles (no vnodes or
file descriptors).
--> This does imply that the code does not know if/when a file is deleted,
      so the Opens/Locks can persist after the file is deleted. My understanding
      is that the NFSv4 client is responsible for doing LockU/Close operations.
      (You could argue that there should be some sort of garbage collection
       running in background to get rid of Opens/Locks that exist after the
       file has been deleted, but the current code does not do that. The
       counter argument is that the client should be doing LockU/Close
       and the structures are pretty harmless.)

Btw, I believe the Linux knfsd server does hold some sort of reference
to the file for Opens/Locks and since the Linux client mainly gets tested
against the Linux knfsd, they might have found that LockU/Close's are
not needed for the Linux knfsd. (This is just conjecture.)

rick

>
> Best regards,
>
> --
> Paul Barker
>