Re: nfs server issues

Paul Barker <[email protected]> Tue, 07 Jul 2026 13:30:17 +0100
Newsgroups	gmane.os.freebsd.devel.file-systems
Message-ID	<[email protected]>
On Tue, 2026-07-07 at 14:29 +0300, Konstantin Belousov wrote:
> On Tue, Jul 07, 2026 at 12:18:21PM +0100, Paul Barker wrote:
> > On Tue, 2026-07-07 at 09:52 +0100, Richard Purdie wrote:
> > > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> > > > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> > > > <[email protected]> wrote:
> > > > > 
> > > > > nfsstat -E -s
> > > > > Server Info:
> > > > >       Getattr      Setattr       Lookup     Readlink         Read        Write
> > > > >   17354100116   1674996998   5136769057        40544  18429670397    986443058
> > > > >        Create       Remove       Rename         Link      Symlink        Mkdir
> > > > >             0     98646613    119865835     17071853            0            0
> > > > >         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
> > > > >             0   1655544711            0   4129535844            0            0
> > > > >        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
> > > > >             0            0      8723436            0            0            0
> > > > One more thing to note. The # of Commits is much smaller than the # of Writes
> > > > on the top line. This "hints" that a lot of the Writes are being done File_sync.
> > > > (To check that, you'd need to capture packets and look at them in wireshark.)
> > > > 
> > > > If a lot of the Writes are File_sync (which means the server must commit the
> > > > data/metadata changes to stable storage before replying), doing the "mirrored
> > > > pair of storage devices dedicated to the ZIL" could be what you need to get
> > > > writes to work well. (As I mentioned, the "cheat" alternative is to
> > > > set "sync=disabled",
> > > > but that runs a risk of data loss when the NFS server crashes/reboots.)
> > > 
> > > I was able to find a way to increase the Open/Lock counts on the
> > > server. Most of the locking we do on the clients is from a tool called
> > > Bitbake and the locking is isolated to specific function. I maintain
> > > Bitbake and wrote/maintain the locking code over the <too many> years.
> > > I copied and pasted the relevant code straight from bitbake into a test
> > > script:
> > > 
> > > https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py
> > > 
> > > (code is from
> > > https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)
> > > 
> > > Obviously it could easily be simplified more but I wanted to test our
> > > actual code. The lock file name/path is hardcoded at the end of the
> > > script but easily changed.
> > > 
> > > I ran that script on three of the NFS clients for around 5 minutes. On
> > > the server, the counts went from:
> > > 
> > > nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> > > CB                    2      6307        55        55         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> > > CB                    3      6543        77        77         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> > > CB                    2      6571        83        83         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> > > 
> > > to:
> > > 
> > > nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> > > CB                    2      6462        65        65         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> > > CB                    3      6679        83        83         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> > > CB                    2      6691        88        88         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
> > > 
> > > I've then stopped the scripts and waited and the counts just stayed
> > > there.
> > > 
> > > I'd be very interested if someone else could reproduce that and if so,
> > > explain what is happening and if it is an issue? 
> > > 
> > > I can't prove the write hangs are related to the counts but it does
> > > seem there is some issue there regardless as they don't make sense.
> > 
> > Hi all,
> > 
> > This is a long one...
> > 
> > I've looked into this issue with the Yocto Project autobuilder cluster
> > alongside Richard. I've used Claude to help me navigate the Linux and
> > FreeBSD source trees, along with manually checking some of the key
> > findings, so this is in the territory of a "working theory" as to what's
> > going on. Some input from folks more familiar with the FreeBSD NFS
> > server code would definitely be helpful to confirm I'm on the right
> > track.
> > 
> > The issue we're seeing is probably triggered by the following sequence
> > of events:
> > 
> > 1) Client A opens a file on the FreeBSD server via an NFS mount. Even if
> >    the client immediately closes the file, Linux's NFS client code will
> >    keep the file handle cached for a short while before sending a CLOSE
> >    to the server.
> > 
> > 2) During this window, client B deletes or replaces the file. This sends
> >    a REMOVE or RENAME message to the server. The FreeBSD NFS server code
> >    unlinks the underlying file immediately.
> > 
> > 3) Client A now releases the file, either explicitly or due to cache
> >    timing out. At this point it sends a PUTFH/CLOSE compound message to
> >    the NFS server, as CLOSE acts on the current file handle the PUTFH is
> >    needed to select which file is being closed. However, the file handle
> >    is now stale due to the removal from another client so the PUTFH
> >    operation fails with ESTALE. The CLOSE operation is never reached, so
> >    no resource cleanup occurs. The file handle has now leaked and will
> >    not be released unless the client session which owns it terminates.
> > 
> > I've confirmed that the file becomes stale with a simple experiment:
> > 
> > 1) Open a file via a Python shell on client A. Leave this Python shell
> >    running.
> > 
> >     >>> fh = open("/srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt", "r")
> > 
> > 2) Remove the file from client B:
> > 
> >     $ rm /srv/autobuilder/valkyrie.yocto.io/pub/non-release/testfile.txt
> > 
> > 3) Now attempt to read the file via the open handle on client A:
> > 
> >     >>> fh.read()
> >     Traceback (most recent call last):
> >       File "<python-input-2>", line 1, in <module>
> >         fh.read()
> >         ~~~~~~~^^
> >     OSError: [Errno 116] Stale file handle
> > 
> > I've also looked at the NFS server code myself on the freebsd-src
> > stable/14 branch. In nfsrvd_compound(), case NFSV4OP_PUTFH calls
> > nfsd_fhtovp(), which calls nfsvno_fhtovp() to resolve the file handle.
> > This sets nd->nd_repstat to ESTALE if the file has been unlinked. This
> > causes a break from the main loop in nfsrvd_compound() to return the
> > error to the client, so the CLOSE operation is never processed. And if
> > CLOSE is never processed, the open file handle leaks.
> > 
> > This causes us two kinds of problems:
> > 
> > 1) The hash table of open file handles grows. With >400k file handles in
> >    the hash table we see the CPU load over 5000%. There is one file
> >    handle table in the NFS server, with one lock, and all stateful
> >    operations seem to need to take this lock to lookup a file handle, so
> >    this leads to thrashing.
> > 
> >    Linux may interpret timeouts waiting for a server response as
> >    EREMOTEIO.
> > 
> >    This can be somewhat ameliorated by increasing vfs.nfsd.fhhashsize to
> >    spread the open handles across more buckets in the hash table, but
> >    that just delays the load issues, it doesn't resolve them.
> > 
> > 2) When the total number of open file handles + open locks hits 500,000,
> >    we hit the default v4statelimit. This causes the server to reply to
> >    operations with NFSERR_RESOURCE which Linux also interprets as
> >    EREMOTEIO.
> > 
> >    The ~483,000 open file handles in Richard's original message is
> >    consistent with the 500k limit being reached before a few clients
> >    were disconnected (zeroing out their open counts and reducing the
> >    overall total).
> > 
> >    We could increase vfs.nfsd.v4statelimit, but again this is just
> >    delaying the inevitable issues. If we're leaking ~500k open file
> >    handles in less than a week then no limit will last very long.
> > 
> > So, essentially we would need to avoid the leak in the first place.
> > Should the error handling for PUTFH on a stale file handle free the
> > associated state? Or should a REMOVE/RENAME operation leave the file
> > handle intact until all consumers have released the file handle (this is
> > what a Linux NFS server seems to do)?
> > 
> > Thank you for bearing with that long explanation! If I've misunderstood
> > anything then please let me know and I'll take another look.
> 
> Does the FreeBSD NFS server still reference the vnode for the deleted file
> in your situation?  If yes, then c041b82c59e885d106f45454302bad3d9dcc58a7
> might already help.

Hi Konstantin, thanks for the info. I sadly don't think this patch will
help, we're using ZFS as the backing store for these NFS shares and that
commit only changes UFS/FFS code. I also don't think the NFS server
holds a vnode reference for each open stateid but I may be wrong, I'm
not really familiar with the code base.

Best regards,

-- 
Paul Barker
signature.asc (application/pgp-signature, 252 B)