Re: nfs server issues

On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> <[email protected]> wrote:
> > 
> > nfsstat -E -s
> > Server Info:
> >       Getattr      Setattr       Lookup     Readlink         Read        Write
> >   17354100116   1674996998   5136769057        40544  18429670397    986443058
> >        Create       Remove       Rename         Link      Symlink        Mkdir
> >             0     98646613    119865835     17071853            0            0
> >         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
> >             0   1655544711            0   4129535844            0            0
> >        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
> >             0            0      8723436            0            0            0
> One more thing to note. The # of Commits is much smaller than the # of Writes
> on the top line. This "hints" that a lot of the Writes are being done File_sync.
> (To check that, you'd need to capture packets and look at them in wireshark.)
> 
> If a lot of the Writes are File_sync (which means the server must commit the
> data/metadata changes to stable storage before replying), doing the "mirrored
> pair of storage devices dedicated to the ZIL" could be what you need to get
> writes to work well. (As I mentioned, the "cheat" alternative is to
> set "sync=disabled",
> but that runs a risk of data loss when the NFS server crashes/reboots.)

I was able to find a way to increase the Open/Lock counts on the
server. Most of the locking we do on the clients is from a tool called
Bitbake and the locking is isolated to specific function. I maintain
Bitbake and wrote/maintain the locking code over the <too many> years.
I copied and pasted the relevant code straight from bitbake into a test
script:

https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py

(code is from
https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)

Obviously it could easily be simplified more but I wanted to test our
actual code. The lock file name/path is hardcoded at the end of the
script but easily changed.

I ran that script on three of the NFS clients for around 5 minutes. On
the server, the counts went from:

nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
CB                    2      6307        55        55         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
CB                    3      6543        77        77         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
CB                    2      6571        83        83         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32

to:

nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
CB                    2      6462        65        65         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
CB                    3      6679        83        83         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
CB                    2      6691        88        88         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32

I've then stopped the scripts and waited and the counts just stayed
there.

I'd be very interested if someone else could reproduce that and if so,
explain what is happening and if it is an issue? 

I can't prove the write hangs are related to the counts but it does
seem there is some issue there regardless as they don't make sense.

Cheers,

Richard