Re: nfs server issues

On Tue, Jul 7, 2026 at 4:18 AM Paul Barker <[email protected]> wrote:
>
> On Tue, 2026-07-07 at 09:52 +0100, Richard Purdie wrote:
> > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> > > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> > > <[email protected]> wrote:
> > > >
> > > > nfsstat -E -s
> > > > Server Info:
> > > >       Getattr      Setattr       Lookup     Readlink         Read        Write
> > > >   17354100116   1674996998   5136769057        40544  18429670397    986443058
> > > >        Create       Remove       Rename         Link      Symlink        Mkdir
> > > >             0     98646613    119865835     17071853            0            0
> > > >         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
> > > >             0   1655544711            0   4129535844            0            0
> > > >        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
> > > >             0            0      8723436            0            0            0
> > > One more thing to note. The # of Commits is much smaller than the # of Writes
> > > on the top line. This "hints" that a lot of the Writes are being done File_sync.
> > > (To check that, you'd need to capture packets and look at them in wireshark.)
> > >
> > > If a lot of the Writes are File_sync (which means the server must commit the
> > > data/metadata changes to stable storage before replying), doing the "mirrored
> > > pair of storage devices dedicated to the ZIL" could be what you need to get
> > > writes to work well. (As I mentioned, the "cheat" alternative is to
> > > set "sync=disabled",
> > > but that runs a risk of data loss when the NFS server crashes/reboots.)
> >
> > I was able to find a way to increase the Open/Lock counts on the
> > server. Most of the locking we do on the clients is from a tool called
> > Bitbake and the locking is isolated to specific function. I maintain
> > Bitbake and wrote/maintain the locking code over the <too many> years.
> > I copied and pasted the relevant code straight from bitbake into a test
> > script:
> >
> > https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py
> >
> > (code is from
> > https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)
> >
> > Obviously it could easily be simplified more but I wanted to test our
> > actual code. The lock file name/path is hardcoded at the end of the
> > script but easily changed.
> >
> > I ran that script on three of the NFS clients for around 5 minutes. On
> > the server, the counts went from:
> >
> > nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> > CB                    2      6307        55        55         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> > CB                    3      6543        77        77         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> > CB                    2      6571        83        83         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> >
> > to:
> >
> > nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> > CB                    2      6462        65        65         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> > CB                    3      6679        83        83         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> > CB                    2      6691        88        88         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> >
> > I've then stopped the scripts and waited and the counts just stayed
> > there.
> >
> > I'd be very interested if someone else could reproduce that and if so,
> > explain what is happening and if it is an issue?
> >
> > I can't prove the write hangs are related to the counts but it does
> > seem there is some issue there regardless as they don't make sense.
>
> Hi all,
>
> This is a long one...
>
> I've looked into this issue with the Yocto Project autobuilder cluster
> alongside Richard. I've used Claude to help me navigate the Linux and
> FreeBSD source trees, along with manually checking some of the key
> findings, so this is in the territory of a "working theory" as to what's
> going on. Some input from folks more familiar with the FreeBSD NFS
> server code would definitely be helpful to confirm I'm on the right
> track.
>
> The issue we're seeing is probably triggered by the following sequence
> of events:
>
> 1) Client A opens a file on the FreeBSD server via an NFS mount. Even if
>    the client immediately closes the file, Linux's NFS client code will
>    keep the file handle cached for a short while before sending a CLOSE
>    to the server.
>
> 2) During this window, client B deletes or replaces the file. This sends
>    a REMOVE or RENAME message to the server. The FreeBSD NFS server code
>    unlinks the underlying file immediately.
>
> 3) Client A now releases the file, either explicitly or due to cache
>    timing out. At this point it sends a PUTFH/CLOSE compound message to
>    the NFS server, as CLOSE acts on the current file handle the PUTFH is
>    needed to select which file is being closed. However, the file handle
>    is now stale due to the removal from another client so the PUTFH
>    operation fails with ESTALE. The CLOSE operation is never reached, so
>    no resource cleanup occurs. The file handle has now leaked and will
>    not be released unless the client session which owns it terminates.
This is certainly possible (and basically what I suggested in another reply
I just did.
If a packet trace shows a PutFH before a Close replying ESTALE, I am not
sure what the server can do?
The PUTFH cannot succeed if the file no longer exists, so it must fail with
ESTALE. The compound must fail when an operation in the compound fails,
so it cannot process the Close.
(The current code does not pre-process the compound RPC before executing
each operation-->it does not know that the CFH set by the PUTFH is only for
a Close.)
--> To be honest, a Close should not require a CFH and should be done
     based entirely on the stateid argument, but that is not what the RFCs
     specify.

As I noted, there might be an argument for "garbage collection" of Opens/Locks
for the case where the file no longer exists, but that is a bunch of
overhead and
I don't think the leaked Opens matter much.

Btw, if you do want to get rid of the leaked state without rebooting
the NFS server,
you can:
- Build a custom kernel without "options NFSD" in it, then...
# service nfsd stop
# kldunload nfsd
# service nfsd start
--> The kldunload will get rid of all the state. The above will look like a
     reboot to the NFS clients, but avoids a full reboot.

Having said all of the above, Writes, particularly ones with a File_stable
argument are much more overhead that leaked Opens.
I suspect your write problem might be related to Write/File_stable. You
can see those in a packet trace.

rick

>
> I've confirmed that the file becomes stale with a simple experiment:
>
> 1) Open a file via a Python shell on client A. Leave this Python shell
>    running.
>
>     >>> fh = open("/srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt", "r")
>
> 2) Remove the file from client B:
>
>     $ rm /srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt
>
> 3) Now attempt to read the file via the open handle on client A:
>
>     >>> fh.read()
>     Traceback (most recent call last):
>       File "<python-input-2>", line 1, in <module>
>         fh.read()
>         ~~~~~~~^^
>     OSError: [Errno 116] Stale file handle
>
> I've also looked at the NFS server code myself on the freebsd-src
> stable/14 branch. In nfsrvd_compound(), case NFSV4OP_PUTFH calls
> nfsd_fhtovp(), which calls nfsvno_fhtovp() to resolve the file handle.
> This sets nd->nd_repstat to ESTALE if the file has been unlinked. This
> causes a break from the main loop in nfsrvd_compound() to return the
> error to the client, so the CLOSE operation is never processed. And if
> CLOSE is never processed, the open file handle leaks.
>
> This causes us two kinds of problems:
>
> 1) The hash table of open file handles grows. With >400k file handles in
>    the hash table we see the CPU load over 5000%. There is one file
>    handle table in the NFS server, with one lock, and all stateful
>    operations seem to need to take this lock to lookup a file handle, so
>    this leads to thrashing.
>
>    Linux may interpret timeouts waiting for a server response as
>    EREMOTEIO.
>
>    This can be somewhat ameliorated by increasing vfs.nfsd.fhhashsize to
>    spread the open handles across more buckets in the hash table, but
>    that just delays the load issues, it doesn't resolve them.
>
> 2) When the total number of open file handles + open locks hits 500,000,
>    we hit the default v4statelimit. This causes the server to reply to
>    operations with NFSERR_RESOURCE which Linux also interprets as
>    EREMOTEIO.
>
>    The ~483,000 open file handles in Richard's original message is
>    consistent with the 500k limit being reached before a few clients
>    were disconnected (zeroing out their open counts and reducing the
>    overall total).
>
>    We could increase vfs.nfsd.v4statelimit, but again this is just
>    delaying the inevitable issues. If we're leaking ~500k open file
>    handles in less than a week then no limit will last very long.
>
> So, essentially we would need to avoid the leak in the first place.
> Should the error handling for PUTFH on a stale file handle free the
> associated state? Or should a REMOVE/RENAME operation leave the file
> handle intact until all consumers have released the file handle (this is
> what a Linux NFS server seems to do)?
>
> Thank you for bearing with that long explanation! If I've misunderstood
> anything then please let me know and I'll take another look.
>
> Best regards,
>
> --
> Paul Barker
>