Re: nfs server issues

On Sat, Jul 4, 2026 at 6:18 AM Rick Macklem <[email protected]> wrote:
>
> On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> <[email protected]> wrote:
> >
> > On Fri, 2026-07-03 at 20:22 -0700, Rick Macklem wrote:
> > > Another thing you can do that might give you a hint w.r.t. what is
> > > going on is to repeatedly do..
> > > # nfsstat -E -s
> > > - and look to see what operations are being done frequently.
> > > (If you see lots of FreeStateID's happening, I think that is a known
> > > bug in the Linux client that was only fixed fairly recently, for
> > > example.)
> >
> > I've pasted the current numbers after the system reset below. What
> > would be considered "lots"? :)
> >
> > Unfortunately we're going to struggle to change the linux kernels
> > connecting to this as those differing versions are part of our test
> > setup.
> >
> > Cheers,
> >
> > Richard
> >
> > nfsstat -E -s
If you "nfsstat -E -s -z >> stats.log" in a loop, doing it once/sec or once
every few seconds and then run that when the server misbehaves, you
might see some ops. ratcheting up quickly, which would give you a hint
w.r.t. what is causing the problem?
(It's more the rate of the ops than the # of them that is useful.)

rick

> > Server Info:
> >       Getattr      Setattr       Lookup     Readlink         Read        Write
> >   17354100116   1674996998   5136769057        40544  18429670397    986443058
> >        Create       Remove       Rename         Link      Symlink        Mkdir
> >             0     98646613    119865835     17071853            0            0
> >         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
> >             0   1655544711            0   4129535844            0            0
> >        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
> >             0            0      8723436            0            0            0
> >          Open     OpenAttr    OpenDwnGr     OpenCfrm    DelePurge       DelRet
> >   33736862240            0           16            0            0            0
> >         GetFH         Lock        LockT        LockU        Close       Verify
> >    3212316334    620514754            0     91450845   1343565521            0
> You'll notice that the # of Opens is far higher than # of Closes, but
> that is normal,
> since an Open can be done to upgrade from ro->rw and that does not result in
> a new Open StateID.
>
> >       NVerify        PutFH     PutPubFH    PutRootFH        Renew    RestoreFH
> >             0  72840514777            0         2939            0     17071831
> >        SaveFH      Secinfo   RelLockOwn     V4Create
> >     136947415            0            0       771785
> > BackChannelCtBindConnToSes   ExchangeID   CreateSess  DestroySess  FreeStateID
> >             0           45         2951         1538         1437     87475206
> >   GetDirDeleg   GetDevInfo   GetDevList layoutCommit    LayoutGet LayoutReturn
> >           468            0            0            0            0            0
> >  SecInfNoName     Sequence       SetSSV  TestStateID    WantDeleg  DestroyClId
> >          1445  72791198609            0         1318            0         9210
> >  ReclaimCompl
> >          1489
> >      Allocate         Copy   CopyNotify   Deallocate     IOAdvise  LayoutError
> >             0         9713            0         4666            0            0
> >   LayoutStats  OffloadCncl  OffloadStat     ReadPlus         Seek    WriteSame
> >             0            0            0          112        48469            0
> >         Clone   GetExtattr   SetExtattr  ListExtattr    RmExtattr
> >            14            0            0        65153            0
> > Server:
> >       Clients    OpenOwner        Opens    LockOwner        Locks       Delegs
> >            44           50        15677          186          186            0
> Having 15K Opens hanging around shouldn't be a big issue. (A NFSv4.2 Open
> is not a POSIX open, it is a form of Windows lock done by Windows when an
> open is done.) Other than it being a fairly busy server, nothing
> really stands out
> when I look at the above.
>
> Having said that, increasing the hash table sizes cannot hurt. (The FreeBSD
> defaults assume the smallest system with minimal resources, like an arm7
> board.)
>
> To do so, add lines like:
> vfs.nfsd.sessionhashsize=N
> vfs.nfsd.clienthashsize=N
> vfs.nfsd.statehashsize=N
> vfs.nfsd.fhhashsize=N
> to the server's /boot/loader.conf.
> (Bumping vfs.nfsd.clienthashsize doesn't make much sense, since you
> only have 44 of them.)
> --> They take affect when the server is rebooted. (Having the other 3
>      in the thousands is harmless.)
>
> Also, if you
> # nfsstat -m
> on the clients, you should see what is actually being used w.r.t.
> mount options.
> In particular, avoid "soft" and "intr" if at all possible. (The dangling
> Opens might be because an RPC doing the Close times out or get
> terminated by a signal in the client?)
> As for rsize/wsize. Larger values improve per client performance,
> but could increase mbuf cluster usage and server load. (The default
> should be 128K, which is the basic ZFS record size. If it is larger
> than that, you might want to try dropping it to 128K by either setting
> nfs_server_maxio in /etc/rc.conf to 128K (takes affect when rebooted)
> or via the wsize mount option in the clients.
>
> The mystery for me is why writes stop working?
> Maybe running "top" when the server is misbehaving will give you
> a hint? (It shows things like wired memory.) Doing
> # vmstat -z | fgrep mbuf
> will show you how many mbufs are being used. If you are running
> out of mbuf clusters, that could cause write problems.
>
> Good luck with it, rick
>
> >       Layouts
> >             0
> > Server Cache Stats:
> >        Inprog     Non-idem       Misses    CacheSize      TCPPeak
> >             0            0            0            0            0