Re: nfs server issues

On Fri, Jul 10, 2026 at 1:01=E2=80=AFPM Michael Halstead
<[email protected]> wrote:
>
>
>
> On Wed, Jul 8, 2026 at 2:38=E2=80=AFPM Rick Macklem <[email protected]=
om> wrote:
>>
>> On Wed, Jul 8, 2026 at 7:57=E2=80=AFAM Rick Macklem <rick.macklem@gmail.=
com> wrote:
>> >
>> > On Wed, Jul 8, 2026 at 2:28=E2=80=AFAM Richard Purdie
>> > <[email protected]> wrote:
>> > >
>> > > On Tue, 2026-07-07 at 15:54 -0700, Michael Halstead wrote:
>> > > > On Tue, Jul 7, 2026 at 1:52=E2=80=AFAM Richard Purdie <richard.pur=
[email protected]> wrote:
>> > > > > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
>> > > > > > On Sat, Jul 4, 2026 at 1:05=E2=80=AFAM Richard Purdie
>> > > > > > <[email protected]> wrote:
>> > > > > > >
>> > > > > > > nfsstat -E -s
>> > > > > > > Server Info:
>> > > > > > >       Getattr      Setattr       Lookup     Readlink        =
 Read        Write
>> > > > > > >   17354100116   1674996998   5136769057        40544  184296=
70397    986443058
>> > > > > > >        Create       Remove       Rename         Link      Sy=
mlink        Mkdir
>> > > > > > >             0     98646613    119865835     17071853        =
    0            0
>> > > > > > >         Rmdir      Readdir     RdirPlus       Access        =
Mknod       Fsstat
>> > > > > > >             0   1655544711            0   4129535844        =
    0            0
>> > > > > > >        FSinfo     pathConf       Commit      LookupP      Se=
tClId    SetClIdCf
>> > > > > > >             0            0      8723436            0        =
    0            0
>> > > > > > One more thing to note. The # of Commits is much smaller than =
the # of Writes
>> > > > > > on the top line. This "hints" that a lot of the Writes are bei=
ng done File_sync.
>> > > > > > (To check that, you'd need to capture packets and look at them=
 in wireshark.)
>> > > >
>> > > >
>> > > > I've captured data
>> > > > at https://valkyrie.yocto.io/pub/non-release/mtest/nfs_capture2.pc=
ap.
>> > > > zst from a fresh reboot running the testlock.py for about 100MB of
>> > > > data and umounting the filesystems. Please let me know if this
>> > > > reveals the issue or what else I can offer.
>> > >
>> > > Thanks Michael. I think that didn't have enough parallel test script=
s
>> > > running. This is one I just captured with more parallelism (4 client=
s),
>> > > just tracing one of them:
>> > >
>> > > https://valkyrie.yocto.io/pub/non-release/mtest/rp_capture.pcap.zst
>> > >
>> > > This one has a lot of NFS4ERR_STALE in it which I suspect is what we
>> > > were looking for...
>> > Yep. This one definitely shows RPCs that would do Close failing when
>> > PutFH replies NFS4ERR_STALE. (I looked at the previous one and couldn'=
t
>> > see any problems in it, although the previous one does have the weird =
case
>> > where the same Lock is acquired twice. Both succeed because it is the
>> > same client and lock_owner.  I am going to check to see what the RFCs
>> > say w.r.t. duplicate lock operations. Like Open, the Lock ops were mod=
elled
>> > after what Windows wants, but were intended to handle POSIX lock seman=
tics
>> > as well. Of course, no one implements mandatory locking and the protoc=
ol
>> > doesn't even allow the client to figure out which the server is doing,=
 but
>> > that's another story.;-)
>> >
>> > I do think I can come up with a simple solution for this:
>> > - Since the operations that get rid of state all require a CFH, which
>> >   must be set by PutFH.. I cannot see any reason to keep state once
>> >   the PutFH is returning NFS4ERR_STALE for the file.
>> >   --> So, I think I can just free all the state for that FH at that ti=
me.
>> >       (The state is keyed on FH and not a vnode, so the PutFH has
>> >        the required FH.)
>> > --> I'll post a patch here later to-day. Hopefully you guys can run a
>> >       kernel from patched sources to test it.
>> I've attached a patch that I think fixes this (minimally tested).
>> If you can test this by applying it to your kernel sources and building =
a
>> kernel from sources, please do so and let us know how it goes.
>>
>> If you've never done this, the following should work (assuming the
>> source tree is /usr/src).
>> # cd /usr/src
>> # patch -p0 < freefh.patch
>> # make buildkernel
>> # make installkernel
>> - reboot
>>
>> rick
>>
>
> I updated to FreeBSD 15.1p1 and applied this patch. Running our testlock.=
py script on 40+ clients at once worked fine. Open counts stayed at 0 or 1 =
for the duration of the test. Our build cluster now has 22 hours uptime wit=
h normal operation and all the stats look great.
>
> nfsstat -E -s
> Server Info:
>       Getattr      Setattr       Lookup     Readlink         Read        =
Write
>     247019884     21674278     77085915            2    145835919      93=
51171
>        Create       Remove       Rename         Link      Symlink        =
Mkdir
>             0      1663599      1990692       278837            0        =
    0
>         Rmdir      Readdir     RdirPlus       Access        Mknod       F=
sstat
>             0      9782666            0     65790267            0        =
    0
>        FSinfo     pathConf       Commit      LookupP      SetClId    SetC=
lIdCf
>             0            0        93580            0            0        =
    0
>          Open     OpenAttr    OpenDwnGr     OpenCfrm    DelePurge       D=
elRet
>      26716944            0            0            0            0        =
    0
>         GetFH         Lock        LockT        LockU        Close       V=
erify
>      47284768     15945046            0      1514379     23213889        =
    0
>       NVerify        PutFH     PutPubFH    PutRootFH        Renew    Rest=
oreFH
>             0    458678547            0           85            0       2=
78837
>        SaveFH      Secinfo   RelLockOwn     V4Create
>       2269529            0            0         8443
> BackChannelCtBindConnToSes   ExchangeID   CreateSess  DestroySess  FreeSt=
ateID
>             0            0           85           47            3      14=
73548
>   GetDirDeleg   GetDevInfo   GetDevList layoutCommit    LayoutGet LayoutR=
eturn
>            26            0            0            0            0        =
    0
>  SecInfNoName     Sequence       SetSSV  TestStateID    WantDeleg  Destro=
yClId
>            41    457883368            0           41            0        =
    0
>  ReclaimCompl
>            44
>      Allocate         Copy   CopyNotify   Deallocate     IOAdvise  Layout=
Error
>             0            0            0            0            0        =
    0
>   LayoutStats  OffloadCncl  OffloadStat     ReadPlus         Seek    Writ=
eSame
>             0            0            0           11          615        =
    0
>         Clone   GetExtattr   SetExtattr  ListExtattr    RmExtattr
>             0            0            0          685            0
> Server:
>       Clients    OpenOwner        Opens    LockOwner        Locks       D=
elegs
>            44            0            0            0            0        =
    0
>       Layouts
>             0
> Server Cache Stats:
>        Inprog     Non-idem       Misses    CacheSize      TCPPeak
>             0            0            0            0            0
>
> When the cluster is idle, `nfsdumpstate` shows zeros across the board as =
expected.
>
> I also added to /boot/loader.conf:
> vfs.nfsd.clienthashsize=3D2048
> vfs.nfsd.sessionhashsize=3D2048
> vfs.nfsd.statehashsize=3D4096
> vfs.nfsd.fhhashsize=3D16384
>
> Are there any other useful detail we can provide?
Not that I can think of.

I'll assume you don't mind being listed as "Tested by:"
for a commit of the patch. (If you do mind, just email me.)

I am still wondering if you will see your write problems,
but I assume it's wait and see for now.

Thanks for testing it, rick
ps: I will probably commit this little patch to main soon. It should be in
      stable/15 and stable/14 for the next releases.

>
>
>>
>> >
>> > Thanks for the packet trace and help tracking this down, rick
>> >
>> > >
>> > > Cheers,
>> > >
>> > > Richard
>
>
>
> --
> Michael Halstead (he/him)
> Linux Foundation / Yocto Project
> Staff IT Engineer