Re: nfs server issues

On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
<[email protected]> wrote:
>
> On Fri, 2026-07-03 at 20:22 -0700, Rick Macklem wrote:
> > Another thing you can do that might give you a hint w.r.t. what is
> > going on is to repeatedly do..
> > # nfsstat -E -s
> > - and look to see what operations are being done frequently.
> > (If you see lots of FreeStateID's happening, I think that is a known
> > bug in the Linux client that was only fixed fairly recently, for
> > example.)
>
> I've pasted the current numbers after the system reset below. What
> would be considered "lots"? :)
>
> Unfortunately we're going to struggle to change the linux kernels
> connecting to this as those differing versions are part of our test
> setup.
>
> Cheers,
>
> Richard
>
> nfsstat -E -s
> Server Info:
>       Getattr      Setattr       Lookup     Readlink         Read        Write
>   17354100116   1674996998   5136769057        40544  18429670397    986443058
>        Create       Remove       Rename         Link      Symlink        Mkdir
>             0     98646613    119865835     17071853            0            0
>         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
>             0   1655544711            0   4129535844            0            0
>        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
>             0            0      8723436            0            0            0
>          Open     OpenAttr    OpenDwnGr     OpenCfrm    DelePurge       DelRet
>   33736862240            0           16            0            0            0
>         GetFH         Lock        LockT        LockU        Close       Verify
>    3212316334    620514754            0     91450845   1343565521            0
You'll notice that the # of Opens is far higher than # of Closes, but
that is normal,
since an Open can be done to upgrade from ro->rw and that does not result in
a new Open StateID.

>       NVerify        PutFH     PutPubFH    PutRootFH        Renew    RestoreFH
>             0  72840514777            0         2939            0     17071831
>        SaveFH      Secinfo   RelLockOwn     V4Create
>     136947415            0            0       771785
> BackChannelCtBindConnToSes   ExchangeID   CreateSess  DestroySess  FreeStateID
>             0           45         2951         1538         1437     87475206
>   GetDirDeleg   GetDevInfo   GetDevList layoutCommit    LayoutGet LayoutReturn
>           468            0            0            0            0            0
>  SecInfNoName     Sequence       SetSSV  TestStateID    WantDeleg  DestroyClId
>          1445  72791198609            0         1318            0         9210
>  ReclaimCompl
>          1489
>      Allocate         Copy   CopyNotify   Deallocate     IOAdvise  LayoutError
>             0         9713            0         4666            0            0
>   LayoutStats  OffloadCncl  OffloadStat     ReadPlus         Seek    WriteSame
>             0            0            0          112        48469            0
>         Clone   GetExtattr   SetExtattr  ListExtattr    RmExtattr
>            14            0            0        65153            0
> Server:
>       Clients    OpenOwner        Opens    LockOwner        Locks       Delegs
>            44           50        15677          186          186            0
Having 15K Opens hanging around shouldn't be a big issue. (A NFSv4.2 Open
is not a POSIX open, it is a form of Windows lock done by Windows when an
open is done.) Other than it being a fairly busy server, nothing
really stands out
when I look at the above.

Having said that, increasing the hash table sizes cannot hurt. (The FreeBSD
defaults assume the smallest system with minimal resources, like an arm7
board.)

To do so, add lines like:
vfs.nfsd.sessionhashsize=N
vfs.nfsd.clienthashsize=N
vfs.nfsd.statehashsize=N
vfs.nfsd.fhhashsize=N
to the server's /boot/loader.conf.
(Bumping vfs.nfsd.clienthashsize doesn't make much sense, since you
only have 44 of them.)
--> They take affect when the server is rebooted. (Having the other 3
     in the thousands is harmless.)

Also, if you
# nfsstat -m
on the clients, you should see what is actually being used w.r.t.
mount options.
In particular, avoid "soft" and "intr" if at all possible. (The dangling
Opens might be because an RPC doing the Close times out or get
terminated by a signal in the client?)
As for rsize/wsize. Larger values improve per client performance,
but could increase mbuf cluster usage and server load. (The default
should be 128K, which is the basic ZFS record size. If it is larger
than that, you might want to try dropping it to 128K by either setting
nfs_server_maxio in /etc/rc.conf to 128K (takes affect when rebooted)
or via the wsize mount option in the clients.

The mystery for me is why writes stop working?
Maybe running "top" when the server is misbehaving will give you
a hint? (It shows things like wired memory.) Doing
# vmstat -z | fgrep mbuf
will show you how many mbufs are being used. If you are running
out of mbuf clusters, that could cause write problems.

Good luck with it, rick

>       Layouts
>             0
> Server Cache Stats:
>        Inprog     Non-idem       Misses    CacheSize      TCPPeak
>             0            0            0            0            0