Re: nfs server issues

Paul Barker <[email protected]> Tue, 07 Jul 2026 12:18:21 +0100
Newsgroups	gmane.os.freebsd.devel.file-systems
Message-ID	<[email protected]>
On Tue, 2026-07-07 at 09:52 +0100, Richard Purdie wrote:
> On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> > <[email protected]> wrote:
> > > 
> > > nfsstat -E -s
> > > Server Info:
> > >       Getattr      Setattr       Lookup     Readlink         Read        Write
> > >   17354100116   1674996998   5136769057        40544  18429670397    986443058
> > >        Create       Remove       Rename         Link      Symlink        Mkdir
> > >             0     98646613    119865835     17071853            0            0
> > >         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
> > >             0   1655544711            0   4129535844            0            0
> > >        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
> > >             0            0      8723436            0            0            0
> > One more thing to note. The # of Commits is much smaller than the # of Writes
> > on the top line. This "hints" that a lot of the Writes are being done File_sync.
> > (To check that, you'd need to capture packets and look at them in wireshark.)
> > 
> > If a lot of the Writes are File_sync (which means the server must commit the
> > data/metadata changes to stable storage before replying), doing the "mirrored
> > pair of storage devices dedicated to the ZIL" could be what you need to get
> > writes to work well. (As I mentioned, the "cheat" alternative is to
> > set "sync=disabled",
> > but that runs a risk of data loss when the NFS server crashes/reboots.)
> 
> I was able to find a way to increase the Open/Lock counts on the
> server. Most of the locking we do on the clients is from a tool called
> Bitbake and the locking is isolated to specific function. I maintain
> Bitbake and wrote/maintain the locking code over the <too many> years.
> I copied and pasted the relevant code straight from bitbake into a test
> script:
> 
> https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py
> 
> (code is from
> https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)
> 
> Obviously it could easily be simplified more but I wanted to test our
> actual code. The lock file name/path is hardcoded at the end of the
> script but easily changed.
> 
> I ran that script on three of the NFS clients for around 5 minutes. On
> the server, the counts went from:
> 
> nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> CB                    2      6307        55        55         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> CB                    3      6543        77        77         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> CB                    2      6571        83        83         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> 
> to:
> 
> nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> CB                    2      6462        65        65         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> CB                    3      6679        83        83         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> CB                    2      6691        88        88         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> 
> I've then stopped the scripts and waited and the counts just stayed
> there.
> 
> I'd be very interested if someone else could reproduce that and if so,
> explain what is happening and if it is an issue? 
> 
> I can't prove the write hangs are related to the counts but it does
> seem there is some issue there regardless as they don't make sense.

Hi all,

This is a long one...

I've looked into this issue with the Yocto Project autobuilder cluster
alongside Richard. I've used Claude to help me navigate the Linux and
FreeBSD source trees, along with manually checking some of the key
findings, so this is in the territory of a "working theory" as to what's
going on. Some input from folks more familiar with the FreeBSD NFS
server code would definitely be helpful to confirm I'm on the right
track.

The issue we're seeing is probably triggered by the following sequence
of events:

1) Client A opens a file on the FreeBSD server via an NFS mount. Even if
   the client immediately closes the file, Linux's NFS client code will
   keep the file handle cached for a short while before sending a CLOSE
   to the server.

2) During this window, client B deletes or replaces the file. This sends
   a REMOVE or RENAME message to the server. The FreeBSD NFS server code
   unlinks the underlying file immediately.

3) Client A now releases the file, either explicitly or due to cache
   timing out. At this point it sends a PUTFH/CLOSE compound message to
   the NFS server, as CLOSE acts on the current file handle the PUTFH is
   needed to select which file is being closed. However, the file handle
   is now stale due to the removal from another client so the PUTFH
   operation fails with ESTALE. The CLOSE operation is never reached, so
   no resource cleanup occurs. The file handle has now leaked and will
   not be released unless the client session which owns it terminates.

I've confirmed that the file becomes stale with a simple experiment:

1) Open a file via a Python shell on client A. Leave this Python shell
   running.

    >>> fh = open("/srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt", "r")

2) Remove the file from client B:

    $ rm /srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt

3) Now attempt to read the file via the open handle on client A:

    >>> fh.read()
    Traceback (most recent call last):
      File "<python-input-2>", line 1, in <module>
        fh.read()
        ~~~~~~~^^
    OSError: [Errno 116] Stale file handle

I've also looked at the NFS server code myself on the freebsd-src
stable/14 branch. In nfsrvd_compound(), case NFSV4OP_PUTFH calls
nfsd_fhtovp(), which calls nfsvno_fhtovp() to resolve the file handle.
This sets nd->nd_repstat to ESTALE if the file has been unlinked. This
causes a break from the main loop in nfsrvd_compound() to return the
error to the client, so the CLOSE operation is never processed. And if
CLOSE is never processed, the open file handle leaks.

This causes us two kinds of problems:

1) The hash table of open file handles grows. With >400k file handles in
   the hash table we see the CPU load over 5000%. There is one file
   handle table in the NFS server, with one lock, and all stateful
   operations seem to need to take this lock to lookup a file handle, so
   this leads to thrashing.

   Linux may interpret timeouts waiting for a server response as
   EREMOTEIO.

   This can be somewhat ameliorated by increasing vfs.nfsd.fhhashsize to
   spread the open handles across more buckets in the hash table, but
   that just delays the load issues, it doesn't resolve them.

2) When the total number of open file handles + open locks hits 500,000,
   we hit the default v4statelimit. This causes the server to reply to
   operations with NFSERR_RESOURCE which Linux also interprets as
   EREMOTEIO.

   The ~483,000 open file handles in Richard's original message is
   consistent with the 500k limit being reached before a few clients
   were disconnected (zeroing out their open counts and reducing the
   overall total).

   We could increase vfs.nfsd.v4statelimit, but again this is just
   delaying the inevitable issues. If we're leaking ~500k open file
   handles in less than a week then no limit will last very long.

So, essentially we would need to avoid the leak in the first place.
Should the error handling for PUTFH on a stale file handle free the
associated state? Or should a REMOVE/RENAME operation leave the file
handle intact until all consumers have released the file handle (this is
what a Linux NFS server seems to do)?

Thank you for bearing with that long explanation! If I've misunderstood
anything then please let me know and I'll take another look.

Best regards,

-- 
Paul Barker
signature.asc (application/pgp-signature, 252 B)