Re: nfs server issues
Michael Halstead <[email protected]> Fri, 10 Jul 2026 13:01:27 -0700
| Newsgroups | gmane.os.freebsd.devel.file-systems |
|---|---|
| Message-ID | <CADfgfobRe1F_szqKxJNj3AhRDSep5h7DEocn87f10ehNbPm5ew@mail.gmail.com> |
--0000000000007470740656473557 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jul 8, 2026 at 2:38=E2=80=AFPM Rick Macklem <[email protected]= > wrote: > On Wed, Jul 8, 2026 at 7:57=E2=80=AFAM Rick Macklem <[email protected]= om> > wrote: > > > > On Wed, Jul 8, 2026 at 2:28=E2=80=AFAM Richard Purdie > > <[email protected]> wrote: > > > > > > On Tue, 2026-07-07 at 15:54 -0700, Michael Halstead wrote: > > > > On Tue, Jul 7, 2026 at 1:52=E2=80=AFAM Richard Purdie < > [email protected]> wrote: > > > > > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote: > > > > > > On Sat, Jul 4, 2026 at 1:05=E2=80=AFAM Richard Purdie > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > nfsstat -E -s > > > > > > > Server Info: > > > > > > > Getattr Setattr Lookup Readlink > Read Write > > > > > > > 17354100116 1674996998 5136769057 40544 > 18429670397 986443058 > > > > > > > Create Remove Rename Link > Symlink Mkdir > > > > > > > 0 98646613 119865835 17071853 > 0 0 > > > > > > > Rmdir Readdir RdirPlus Access > Mknod Fsstat > > > > > > > 0 1655544711 0 4129535844 > 0 0 > > > > > > > FSinfo pathConf Commit LookupP > SetClId SetClIdCf > > > > > > > 0 0 8723436 0 > 0 0 > > > > > > One more thing to note. The # of Commits is much smaller than > the # of Writes > > > > > > on the top line. This "hints" that a lot of the Writes are bein= g > done File_sync. > > > > > > (To check that, you'd need to capture packets and look at them > in wireshark.) > > > > > > > > > > > > I've captured data > > > > at https://valkyrie.yocto.io/pub/non-release/mtest/nfs_capture2.pca= p > . > > > > zst from a fresh reboot running the testlock.py for about 100MB of > > > > data and umounting the filesystems. Please let me know if this > > > > reveals the issue or what else I can offer. > > > > > > Thanks Michael. I think that didn't have enough parallel test scripts > > > running. This is one I just captured with more parallelism (4 clients= ), > > > just tracing one of them: > > > > > > https://valkyrie.yocto.io/pub/non-release/mtest/rp_capture.pcap.zst > > > > > > This one has a lot of NFS4ERR_STALE in it which I suspect is what we > > > were looking for... > > Yep. This one definitely shows RPCs that would do Close failing when > > PutFH replies NFS4ERR_STALE. (I looked at the previous one and couldn't > > see any problems in it, although the previous one does have the weird > case > > where the same Lock is acquired twice. Both succeed because it is the > > same client and lock_owner. I am going to check to see what the RFCs > > say w.r.t. duplicate lock operations. Like Open, the Lock ops were > modelled > > after what Windows wants, but were intended to handle POSIX lock > semantics > > as well. Of course, no one implements mandatory locking and the protoco= l > > doesn't even allow the client to figure out which the server is doing, > but > > that's another story.;-) > > > > I do think I can come up with a simple solution for this: > > - Since the operations that get rid of state all require a CFH, which > > must be set by PutFH.. I cannot see any reason to keep state once > > the PutFH is returning NFS4ERR_STALE for the file. > > --> So, I think I can just free all the state for that FH at that tim= e. > > (The state is keyed on FH and not a vnode, so the PutFH has > > the required FH.) > > --> I'll post a patch here later to-day. Hopefully you guys can run a > > kernel from patched sources to test it. > I've attached a patch that I think fixes this (minimally tested). > If you can test this by applying it to your kernel sources and building a > kernel from sources, please do so and let us know how it goes. > > If you've never done this, the following should work (assuming the > source tree is /usr/src). > # cd /usr/src > # patch -p0 < freefh.patch > # make buildkernel > # make installkernel > - reboot > > rick > > I updated to FreeBSD 15.1p1 and applied this patch. Running our testlock.py script on 40+ clients at once worked fine. Open counts stayed at 0 or 1 for the duration of the test. Our build cluster now has 22 hours uptime with normal operation and all the stats look great. nfsstat -E -s Server Info: Getattr Setattr Lookup Readlink Read Write 247019884 21674278 77085915 2 145835919 9351171 Create Remove Rename Link Symlink Mkdir 0 1663599 1990692 278837 0 0 Rmdir Readdir RdirPlus Access Mknod Fsstat 0 9782666 0 65790267 0 0 FSinfo pathConf Commit LookupP SetClId SetClIdCf 0 0 93580 0 0 0 Open OpenAttr OpenDwnGr OpenCfrm DelePurge DelRet 26716944 0 0 0 0 0 GetFH Lock LockT LockU Close Verify 47284768 15945046 0 1514379 23213889 0 NVerify PutFH PutPubFH PutRootFH Renew RestoreFH 0 458678547 0 85 0 278837 SaveFH Secinfo RelLockOwn V4Create 2269529 0 0 8443 BackChannelCtBindConnToSes ExchangeID CreateSess DestroySess FreeStateID 0 0 85 47 3 1473548 GetDirDeleg GetDevInfo GetDevList layoutCommit LayoutGet LayoutReturn 26 0 0 0 0 0 SecInfNoName Sequence SetSSV TestStateID WantDeleg DestroyClId 41 457883368 0 41 0 0 ReclaimCompl 44 Allocate Copy CopyNotify Deallocate IOAdvise LayoutError 0 0 0 0 0 0 LayoutStats OffloadCncl OffloadStat ReadPlus Seek WriteSame 0 0 0 11 615 0 Clone GetExtattr SetExtattr ListExtattr RmExtattr 0 0 0 685 0 Server: Clients OpenOwner Opens LockOwner Locks Delegs 44 0 0 0 0 0 Layouts 0 Server Cache Stats: Inprog Non-idem Misses CacheSize TCPPeak 0 0 0 0 0 When the cluster is idle, `nfsdumpstate` shows zeros across the board as expected. I also added to /boot/loader.conf: vfs.nfsd.clienthashsize=3D2048 vfs.nfsd.sessionhashsize=3D2048 vfs.nfsd.statehashsize=3D4096 vfs.nfsd.fhhashsize=3D16384 Are there any other useful detail we can provide? > > > > Thanks for the packet trace and help tracking this down, rick > > > > > > > > Cheers, > > > > > > Richard > --=20 Michael Halstead (he/him) Linux Foundation / Yocto Project Staff IT Engineer --0000000000007470740656473557 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Jul 8, = 2026 at 2:38=E2=80=AFPM Rick Macklem <<a href=3D"mailto:rick.macklem@gma= il.com">[email protected]</a>> wrote:<br></div><blockquote class=3D= "gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(2= 04,204,204);padding-left:1ex">On Wed, Jul 8, 2026 at 7:57=E2=80=AFAM Rick M= acklem <<a href=3D"mailto:[email protected]" target=3D"_blank">rick= [email protected]</a>> wrote:<br> ><br> > On Wed, Jul 8, 2026 at 2:28=E2=80=AFAM Richard Purdie<br> > <<a href=3D"mailto:[email protected]" target=3D"_b= lank">[email protected]</a>> wrote:<br> > ><br> > > On Tue, 2026-07-07 at 15:54 -0700, Michael Halstead wrote:<br> > > > On Tue, Jul 7, 2026 at 1:52=E2=80=AFAM Richard Purdie <<a= href=3D"mailto:[email protected]" target=3D"_blank">richa= [email protected]</a>> wrote:<br> > > > > On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:<= br> > > > > > On Sat, Jul 4, 2026 at 1:05=E2=80=AFAM Richard Pur= die<br> > > > > > <<a href=3D"mailto:richard.purdie@linuxfoundati= on.org" target=3D"_blank">[email protected]</a>> wrote:= <br> > > > > > ><br> > > > > > > nfsstat -E -s<br> > > > > > > Server Info:<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0Getattr=C2=A0 =C2= =A0 =C2=A0 Setattr=C2=A0 =C2=A0 =C2=A0 =C2=A0Lookup=C2=A0 =C2=A0 =C2=A0Read= link=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Read=C2=A0 =C2=A0 =C2=A0 =C2=A0 Write= <br> > > > > > >=C2=A0 =C2=A017354100116=C2=A0 =C2=A0167499699= 8=C2=A0 =C2=A05136769057=C2=A0 =C2=A0 =C2=A0 =C2=A0 40544=C2=A0 18429670397= =C2=A0 =C2=A0 986443058<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 Create=C2=A0 =C2= =A0 =C2=A0 =C2=A0Remove=C2=A0 =C2=A0 =C2=A0 =C2=A0Rename=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0Link=C2=A0 =C2=A0 =C2=A0 Symlink=C2=A0 =C2=A0 =C2=A0 =C2= =A0 Mkdir<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A00=C2=A0 =C2=A0 =C2=A098646613=C2=A0 =C2=A0 119865835=C2=A0 =C2=A0 =C2=A0= 17071853=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 0<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Rmdir=C2=A0 = =C2=A0 =C2=A0 Readdir=C2=A0 =C2=A0 =C2=A0RdirPlus=C2=A0 =C2=A0 =C2=A0 =C2= =A0Access=C2=A0 =C2=A0 =C2=A0 =C2=A0 Mknod=C2=A0 =C2=A0 =C2=A0 =C2=A0Fsstat= <br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A00=C2=A0 =C2=A01655544711=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2= =A0 =C2=A04129535844=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 FSinfo=C2=A0 =C2= =A0 =C2=A0pathConf=C2=A0 =C2=A0 =C2=A0 =C2=A0Commit=C2=A0 =C2=A0 =C2=A0 Loo= kupP=C2=A0 =C2=A0 =C2=A0 SetClId=C2=A0 =C2=A0 SetClIdCf<br> > > > > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 8723436= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0<br> > > > > > One more thing to note. The # of Commits is much s= maller than the # of Writes<br> > > > > > on the top line. This "hints" that a lot= of the Writes are being done File_sync.<br> > > > > > (To check that, you'd need to capture packets = and look at them in wireshark.)<br> > > ><br> > > ><br> > > > I've captured data<br> > > > at <a href=3D"https://valkyrie.yocto.io/pub/non-release/mtes= t/nfs_capture2.pcap" rel=3D"noreferrer" target=3D"_blank">https://valkyrie.= yocto.io/pub/non-release/mtest/nfs_capture2.pcap</a>.<br> > > > zst from a fresh reboot running the testlock.py for about 10= 0MB of<br> > > > data and umounting the filesystems. Please let me know if th= is<br> > > > reveals the issue or what else I can offer.<br> > ><br> > > Thanks Michael. I think that didn't have enough parallel test= scripts<br> > > running. This is one I just captured with more parallelism (4 cli= ents),<br> > > just tracing one of them:<br> > ><br> > > <a href=3D"https://valkyrie.yocto.io/pub/non-release/mtest/rp_cap= ture.pcap.zst" rel=3D"noreferrer" target=3D"_blank">https://valkyrie.yocto.= io/pub/non-release/mtest/rp_capture.pcap.zst</a><br> > ><br> > > This one has a lot of NFS4ERR_STALE in it which I suspect is what= we<br> > > were looking for...<br> > Yep. This one definitely shows RPCs that would do Close failing when<b= r> > PutFH replies NFS4ERR_STALE. (I looked at the previous one and couldn&= #39;t<br> > see any problems in it, although the previous one does have the weird = case<br> > where the same Lock is acquired twice. Both succeed because it is the<= br> > same client and lock_owner.=C2=A0 I am going to check to see what the = RFCs<br> > say w.r.t. duplicate lock operations. Like Open, the Lock ops were mod= elled<br> > after what Windows wants, but were intended to handle POSIX lock seman= tics<br> > as well. Of course, no one implements mandatory locking and the protoc= ol<br> > doesn't even allow the client to figure out which the server is do= ing, but<br> > that's another story.;-)<br> ><br> > I do think I can come up with a simple solution for this:<br> > - Since the operations that get rid of state all require a CFH, which<= br> >=C2=A0 =C2=A0must be set by PutFH.. I cannot see any reason to keep sta= te once<br> >=C2=A0 =C2=A0the PutFH is returning NFS4ERR_STALE for the file.<br> >=C2=A0 =C2=A0--> So, I think I can just free all the state for that = FH at that time.<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(The state is keyed on FH and not a vnode, s= o the PutFH has<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 the required FH.)<br> > --> I'll post a patch here later to-day. Hopefully you guys can= run a<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0kernel from patched sources to test it.<br> I've attached a patch that I think fixes this (minimally tested).<br> If you can test this by applying it to your kernel sources and building a<b= r> kernel from sources, please do so and let us know how it goes.<br> <br> If you've never done this, the following should work (assuming the<br> source tree is /usr/src).<br> # cd /usr/src<br> # patch -p0 < freefh.patch<br> # make buildkernel<br> # make installkernel<br> - reboot<br> <br> rick<br> <br></blockquote><div><br></div><div>I updated to FreeBSD 15.1p1 and applie= d this patch. Running our testlock.py script on 40+ clients at once worked = fine. Open counts stayed at 0 or 1 for the duration of the test. Our build = cluster now has 22 hours uptime with normal operation and all the stats loo= k great.</div><div><br></div><div>nfsstat -E -s<br>Server Info:<br>=C2=A0 = =C2=A0 =C2=A0 Getattr =C2=A0 =C2=A0 =C2=A0Setattr =C2=A0 =C2=A0 =C2=A0 Look= up =C2=A0 =C2=A0 Readlink =C2=A0 =C2=A0 =C2=A0 =C2=A0 Read =C2=A0 =C2=A0 = =C2=A0 =C2=A0Write<br>=C2=A0 =C2=A0 247019884 =C2=A0 =C2=A0 21674278 =C2=A0= =C2=A0 77085915 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02 =C2=A0 =C2=A014= 5835919 =C2=A0 =C2=A0 =C2=A09351171<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0Create = =C2=A0 =C2=A0 =C2=A0 Remove =C2=A0 =C2=A0 =C2=A0 Rename =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Link =C2=A0 =C2=A0 =C2=A0Symlink =C2=A0 =C2=A0 =C2=A0 =C2=A0Mkdi= r<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0166359= 9 =C2=A0 =C2=A0 =C2=A01990692 =C2=A0 =C2=A0 =C2=A0 278837 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br>=C2= =A0 =C2=A0 =C2=A0 =C2=A0 Rmdir =C2=A0 =C2=A0 =C2=A0Readdir =C2=A0 =C2=A0 Rd= irPlus =C2=A0 =C2=A0 =C2=A0 Access =C2=A0 =C2=A0 =C2=A0 =C2=A0Mknod =C2=A0 = =C2=A0 =C2=A0 Fsstat<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 = =C2=A0 =C2=A09782666 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2= =A0 65790267 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0FSinfo =C2=A0 =C2= =A0 pathConf =C2=A0 =C2=A0 =C2=A0 Commit =C2=A0 =C2=A0 =C2=A0LookupP =C2=A0= =C2=A0 =C2=A0SetClId =C2=A0 =C2=A0SetClIdCf<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 = =C2=A0 =C2=A093580 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br= >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Open =C2=A0 =C2=A0 OpenAttr =C2=A0 =C2= =A0OpenDwnGr =C2=A0 =C2=A0 OpenCfrm =C2=A0 =C2=A0DelePurge =C2=A0 =C2=A0 = =C2=A0 DelRet<br>=C2=A0 =C2=A0 =C2=A026716944 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 GetFH = =C2=A0 =C2=A0 =C2=A0 =C2=A0 Lock =C2=A0 =C2=A0 =C2=A0 =C2=A0LockT =C2=A0 = =C2=A0 =C2=A0 =C2=A0LockU =C2=A0 =C2=A0 =C2=A0 =C2=A0Close =C2=A0 =C2=A0 = =C2=A0 Verify<br>=C2=A0 =C2=A0 =C2=A047284768 =C2=A0 =C2=A0 15945046 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A01514379 =C2=A0 =C2= =A0 23213889 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br>=C2=A0 =C2=A0 = =C2=A0 NVerify =C2=A0 =C2=A0 =C2=A0 =C2=A0PutFH =C2=A0 =C2=A0 PutPubFH =C2= =A0 =C2=A0PutRootFH =C2=A0 =C2=A0 =C2=A0 =C2=A0Renew =C2=A0 =C2=A0RestoreFH= <br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0458678547 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 85 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 278837<b= r>=C2=A0 =C2=A0 =C2=A0 =C2=A0SaveFH =C2=A0 =C2=A0 =C2=A0Secinfo =C2=A0 RelL= ockOwn =C2=A0 =C2=A0 V4Create<br>=C2=A0 =C2=A0 =C2=A0 2269529 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 8443<br>BackChannelCtBindConnToSes =C2=A0 Excha= ngeID =C2=A0 CreateSess =C2=A0DestroySess =C2=A0FreeStateID<br>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 85 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 47 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03 =C2=A0 =C2=A0 =C2=A01473548<b= r>=C2=A0 GetDirDeleg =C2=A0 GetDevInfo =C2=A0 GetDevList layoutCommit =C2= =A0 =C2=A0LayoutGet LayoutReturn<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A026 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br= >=C2=A0SecInfNoName =C2=A0 =C2=A0 Sequence =C2=A0 =C2=A0 =C2=A0 SetSSV =C2= =A0TestStateID =C2=A0 =C2=A0WantDeleg =C2=A0DestroyClId<br>=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A041 =C2=A0 =C2=A0457883368 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 41 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br>= =C2=A0ReclaimCompl<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A044<br>=C2=A0= =C2=A0 =C2=A0Allocate =C2=A0 =C2=A0 =C2=A0 =C2=A0 Copy =C2=A0 CopyNotify = =C2=A0 Deallocate =C2=A0 =C2=A0 IOAdvise =C2=A0LayoutError<br>=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A00<br>=C2=A0 LayoutStats =C2=A0OffloadCncl =C2=A0Offloa= dStat =C2=A0 =C2=A0 ReadPlus =C2=A0 =C2=A0 =C2=A0 =C2=A0 Seek =C2=A0 =C2=A0= WriteSame<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 11 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0615 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 Clon= e =C2=A0 GetExtattr =C2=A0 SetExtattr =C2=A0ListExtattr =C2=A0 =C2=A0RmExta= ttr<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0685 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br>Serv= er:<br>=C2=A0 =C2=A0 =C2=A0 Clients =C2=A0 =C2=A0OpenOwner =C2=A0 =C2=A0 = =C2=A0 =C2=A0Opens =C2=A0 =C2=A0LockOwner =C2=A0 =C2=A0 =C2=A0 =C2=A0Locks = =C2=A0 =C2=A0 =C2=A0 Delegs<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A044 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00<br>=C2=A0= =C2=A0 =C2=A0 Layouts<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0<br>Se= rver Cache Stats:<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0Inprog =C2=A0 =C2=A0 Non-id= em =C2=A0 =C2=A0 =C2=A0 Misses =C2=A0 =C2=A0CacheSize =C2=A0 =C2=A0 =C2=A0T= CPPeak<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00= <br></div><div><br></div><div>When the cluster is idle, `nfsdumpstate` show= s zeros across the board as expected.</div><div>=C2=A0<br></div><div>I also= added to /boot/loader.conf:</div><div>vfs.nfsd.clienthashsize=3D2048<br>vf= s.nfsd.sessionhashsize=3D2048<br>vfs.nfsd.statehashsize=3D4096<br>vfs.nfsd.= fhhashsize=3D16384<br></div><div><br></div><div>Are there any other useful = detail we can provide?</div><div><br></div><div>=C2=A0</div><blockquote cla= ss=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"> ><br> > Thanks for the packet trace and help tracking this down, rick<br> ><br> > ><br> > > Cheers,<br> > ><br> > > Richard<br> </blockquote></div><div><br clear=3D"all"></div><div><br></div><span class= =3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_s= ignature"><div dir=3D"ltr">Michael Halstead (he/him)<br>Linux Foundation / = Yocto Project<br>Staff IT Engineer</div></div></div> --0000000000007470740656473557--