Re: nfs server issues

On Tue, Jul 7, 2026 at 1:52 AM Richard Purdie
<[email protected]> wrote:
>
> On Sat, 2026-07-04 at 12:41 -0700, Rick Macklem wrote:
> > On Sat, Jul 4, 2026 at 1:05 AM Richard Purdie
> > <[email protected]> wrote:
> > >
> > > nfsstat -E -s
> > > Server Info:
> > >       Getattr      Setattr       Lookup     Readlink         Read        Write
> > >   17354100116   1674996998   5136769057        40544  18429670397    986443058
> > >        Create       Remove       Rename         Link      Symlink        Mkdir
> > >             0     98646613    119865835     17071853            0            0
> > >         Rmdir      Readdir     RdirPlus       Access        Mknod       Fsstat
> > >             0   1655544711            0   4129535844            0            0
> > >        FSinfo     pathConf       Commit      LookupP      SetClId    SetClIdCf
> > >             0            0      8723436            0            0            0
> > One more thing to note. The # of Commits is much smaller than the # of Writes
> > on the top line. This "hints" that a lot of the Writes are being done File_sync.
> > (To check that, you'd need to capture packets and look at them in wireshark.)
> >
> > If a lot of the Writes are File_sync (which means the server must commit the
> > data/metadata changes to stable storage before replying), doing the "mirrored
> > pair of storage devices dedicated to the ZIL" could be what you need to get
> > writes to work well. (As I mentioned, the "cheat" alternative is to
> > set "sync=disabled",
> > but that runs a risk of data loss when the NFS server crashes/reboots.)
>
> I was able to find a way to increase the Open/Lock counts on the
> server. Most of the locking we do on the clients is from a tool called
> Bitbake and the locking is isolated to specific function. I maintain
> Bitbake and wrote/maintain the locking code over the <too many> years.
> I copied and pasted the relevant code straight from bitbake into a test
> script:
>
> https://valkyrie.yocto.io/pub/non-release/rptest/testlock.py
>
> (code is from
> https://git.openembedded.org/bitbake/tree/lib/bb/utils.py)
>
> Obviously it could easily be simplified more but I wanted to test our
> actual code. The lock file name/path is hardcoded at the end of the
> script but easily changed.
>
> I ran that script on three of the NFS clients for around 5 minutes. On
> the server, the counts went from:
>
> nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> CB                    2      6307        55        55         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> CB                    3      6543        77        77         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> CB                    2      6571        83        83         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
>
> to:
>
> nfsdumpstate  | egrep 'fd01:172:16::242:2157|fd01:172:16::12|fd01:172:16:1::11  '
> CB                    2      6462        65        65         0         0 fd01:172:16::242:2157                         4c696e7578204e465376342e322064656269616e31322d766b2d31
> CB                    3      6679        83        83         0         0 fd01:172:16::12                               4c696e7578204e465376342e322064656269616e31322d766b2d33
> CB                    2      6691        88        88         0         0 fd01:172:16:1::11                             4c696e7578204e465376342e322064656269616e31322d766b2d32
>
> I've then stopped the scripts and waited and the counts just stayed
> there.
>
> I'd be very interested if someone else could reproduce that and if so,
> explain what is happening and if it is an issue?
The next step would be to run a test like the above while capturing packets.
# tcpdump -s 0 -w out.pcap host <nfs-client>
on the server should do it.

Then pull out.pcap into wireshark and look for LockU and Close operations.
(If you want, you can stick out.pcap somewhere and I can look at it.)

If the operations are failing (due to ESTALE, for example), then I could
consider allowing these operations to succeed even when the file no
longer exists. Both LockU and Close take a CFH (current file handle)
as arguments, which suggests a valid file handle is required (per the RFC)
but I can see the argument that "the file is gone, so allow it anyhow".

However, if the operations are not in the packet trace, then I'd call
that a client bug, since clients are expected to LockU byte range locks
and Close opens. (There is no server->client callback for Open or Locks.)

I will also make the generic statement...
NFS is not (and will never be, from what I've seen) a POSIX compliant
file system (due to limitations in the protocol design). As such, some POSIX
applications will not work well over NFS.

>
> I can't prove the write hangs are related to the counts but it does
> seem there is some issue there regardless as they don't make sense.
As I noted before, 15K Opens should not have any real effect. They are
just small in-memory data structures on linked lists anchored in hash tables.
(They might result in slightly slower response, due to the time taken for a
linked list traversal. Making the hash tables larger should resolve that and
only results in somewhat bigger tables, which is not a problem, given the
amount of memory in modern systems.)
--> To give you a clue, long ago when I developed this stuff, I tested 100
     NFS clients against a server with only a couple of Gbytes of ram
     (yes, a 32bit x86 around year 2000). I got it up to 500K Opens and it
      still worked).

rick

>
> Cheers,
>
> Richard