Re: nfs server issues

On Tue, Jul 7, 2026 at 1:52 AM Richard Purdie <
[email protected]> wrote:

> On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> > <[email protected]> wrote:
> > >
> > > nfsstat -E -s
> > > Server Info:
> > >       Getattr      Setattr       Lookup     Readlink
> Read        Write
> > >   17354100116   1674996998   5136769057        40544  18429670397
> 986443058
> > >        Create       Remove       Rename         Link
> Symlink        Mkdir
> > >             0     98646613    119865835     17071853
> 0            0
> > >         Rmdir      Readdir     RdirPlus       Access
> Mknod       Fsstat
> > >             0   1655544711            0   4129535844
> 0            0
> > >        FSinfo     pathConf       Commit      LookupP      SetClId
> SetClIdCf
> > >             0            0      8723436            0
> 0            0
> > One more thing to note. The # of Commits is much smaller than the # of
> Writes
> > on the top line. This "hints" that a lot of the Writes are being done
> File_sync.
> > (To check that, you'd need to capture packets and look at them in
> wireshark.)
> >


I've captured data at
https://valkyrie.yocto.io/pub/non-release/mtest/nfs_capture2.pcap.zst from
a fresh reboot running the testlock.py for about 100MB of data and
umounting the filesystems. Please let me know if this reveals the issue or
what else I can offer.

Thank you.

>
> > If a lot of the Writes are File_sync (which means the server must commit
> the
> > data/metadata changes to stable storage before replying), doing the
> "mirrored
> > pair of storage devices dedicated to the ZIL" could be what you need to
> get
> > writes to work well. (As I mentioned, the "cheat" alternative is to
> > set "sync=disabled",
> > but that runs a risk of data loss when the NFS server crashes/reboots.)
>
> I was able to find a way to increase the Open/Lock counts on the
> server. Most of the locking we do on the clients is from a tool called
> Bitbake and the locking is isolated to specific function. I maintain
> Bitbake and wrote/maintain the locking code over the <too many> years.
> I copied and pasted the relevant code straight from bitbake into a test
> script:
>
> https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py
>
> (code is from
> https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)
>
> Obviously it could easily be simplified more but I wanted to test our
> actual code. The lock file name/path is hardcoded at the end of the
> script but easily changed.
>
> I ran that script on three of the NFS clients for around 5 minutes. On
> the server, the counts went from:
>
> nfsdumpstate  | egrep
> 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> CB                    2      6307        55        55         0         0
> fd01:172:16::242:2157
>  4c696e7578204e465376342e322064656269616e31322d766b2d31
> CB                    3      6543        77        77         0         0
> fd01:172:16::12
>  4c696e7578204e465376342e322064656269616e31322d766b2d33
> CB                    2      6571        83        83         0         0
> fd01:172:16:1::11
>  4c696e7578204e465376342e322064656269616e31322d766b2d32
>
> to:
>
> nfsdumpstate  | egrep
> 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> CB                    2      6462        65        65         0         0
> fd01:172:16::242:2157
>  4c696e7578204e465376342e322064656269616e31322d766b2d31
> CB                    3      6679        83        83         0         0
> fd01:172:16::12
>  4c696e7578204e465376342e322064656269616e31322d766b2d33
> CB                    2      6691        88        88         0         0
> fd01:172:16:1::11
>  4c696e7578204e465376342e322064656269616e31322d766b2d32
>
> I've then stopped the scripts and waited and the counts just stayed
> there.
>
> I'd be very interested if someone else could reproduce that and if so,
> explain what is happening and if it is an issue?
>
> I can't prove the write hangs are related to the counts but it does
> seem there is some issue there regardless as they don't make sense.
>
> Cheers,
>
> Richard
>


-- 
Michael Halstead (he/him)
Linux Foundation / Yocto Project
Staff IT Engineer