Re: [nfsv4] Re: simple NFSv4.1/4.2 test of remove while holding a delegation

On Tue, Jun 10, 2025 at 6:38=E2=80=AFAM Dai Ngo
<dai.ngo=3D40oracle.com-Tr9gZwTxerDR74oF6e/[email protected]> wrote:
>
>
> On 6/10/25 6:28 AM, Dai Ngo wrote:
> >
> >
> > On 6/10/25 6:16 AM, Rick Macklem wrote:
> >> On Tue, Jun 10, 2025 at 4:58=E2=80=AFAM Dai Ngo<[email protected]> wr=
ote:
> >>> On 6/9/25 6:06 PM, Rick Macklem wrote:
> >>>> On Mon, Jun 9, 2025 at 5:17=E2=80=AFPM Dai Ngo<[email protected]> w=
rote:
> >>>>> On 6/9/25 4:35 PM, Rick Macklem wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I hope you don't mind a cross-post, but I thought both groups
> >>>>>> might find this interesting...
> >>>>>>
> >>>>>> I have been creating a compound RPC that does REMOVE and
> >>>>>> then tries to determine if the file object has been removed and
> >>>>>> I was surprised to see quite different results from the Linux knfs=
d
> >>>>>> and Solaris 11.4 NFSv4.1/4.2 servers. I think both these servers
> >>>>>> provide FH4_PERSISTENT file handles, although I suppose I
> >>>>>> should check that?
> >>>>>>
> >>>>>> First, the test OPEN/CREATEs a regular file called "foo" (only one
> >>>>>> hard link) and acquires a write delegation for it.
> >>>>>> Then a compound does the following:
> >>>>>> ...
> >>>>>> REMOVE foo
> >>>>>> PUTFH fh for foo
> >>>>>> GETATTR
> >>>>>>
> >>>>>> For the Solaris 11.4 server, the server CB_RECALLs the
> >>>>>> delegation and then replies NFS4ERR_STALE for the PUTFH above.
> >>>>>> (The FreeBSD server currently does the same.)
> >>>>>>
> >>>>>> For a fairly recent Linux (6.12) knfsd, the above replies NFS_OK
> >>>>>> with nlinks =3D=3D 0 in the GETATTR reply.
> >>>>>>
> >>>>>> Hmm. So I've looked in RFC8881 (I'm terrible at reading it so I
> >>>>>> probably missed something) and I cannot find anything that states
> >>>>>> either of the above behaviours is incorrect.
> >>>>>> (NFS4ERR_STALE is listed as an error code for PUTFH, but the
> >>>>>> description of PUTFH only says that it sets the CFH to the fh arg.
> >>>>>> It does not say anything w.r.t. the fh arg. needing to be for a fi=
le
> >>>>>> that still exists.) Neither of these servers sets
> >>>>>> OPEN4_RESULT_PRESERVE_UNLINKED in the OPEN reply.
> >>>>>>
> >>>>>> So, it looks like "file object no longer exists" is indicated eith=
er
> >>>>>> by a NFS4ERR_STALE reply to either PUTFH or GETATTR
> >>>>>> OR
> >>>>>> by a successful reply, but with nlinks =3D=3D 0 for the GETATTR re=
ply.
> >>>>>>
> >>>>>> To be honest, I kinda like the Linux knfsd version, but I am wonde=
ring
> >>>>>> if others think that both of these replies is correct?
> >>>>>>
> >>>>>> Also, is the CB_RECALL needed when the delegation is held by
> >>>>>> the same client as the one doing the REMOVE?
> >>>>> The Linux NFSD detects the delegation belongs to the same client th=
at
> >>>>> causes the conflict (due to REMOVE) and skips the CB_RECALL. This i=
s
> >>>>> an optimization based on the assumption that the client would handl=
e
> >>>>> the conflict locally.
> >>>> And then what does the server do with the delegation?
> >>>> - Does it just discard it, since the file object has been deleted?
> >>>> OR
> >>>> - Does it guarantee that a DELEGRETURN done after the REMOVE will
> >>>>     still work (which seems to be the case for the 6.12 server I am =
using for
> >>>>     testing).
> >>> The delegation remains valid but the file was removed from the namesp=
ace.
> >>> This is why the PUTFH and GETATTR in your test did not fail. However,=
 any
> >>> lookup of the file will fail.
> >>>
> >>>>> If the REMOVE was done by another client, the REMOVE will not compl=
ete
> >>>>> until the delegation is returned. If the PUTFH comes after the REMO=
VE
> >>>>> was completed, it'll  fail with NFS4ERR_STALE since the file, speci=
fied
> >>>>> by the file handle, no longer exists.
> >>>> Assuming the statement w.r.t. "fail with NFS4ERR_STALE" only applies=
 to
> >>>> "REMOVE done by another client" then that sounds fine.
> >>> Correction: even if the REMOVE was done by the another client and the
> >>> delegation was recalled from the 1st client, the open stateid of the =
file
> >>> remains valid until the client sends the CLOSE. So the PUTFH won't fa=
il
> >>> regardless which client sends the REMOVE.
> >> So, should your server be setting OPEN4_RESULT_PRESERVE_UNLINKED
> >> in OPEN replies, given this semantic?
> >> --> If the FH remains valid after REMOVE drops nlink to 0 semantic
> >> were indicated by
> >>       the OPEN4_RESULT_PRESERVE_UNLINKED flag, a client could check fo=
r
> >>       this flag and handle in appropriately.
> > I believe the Linux NFSD currently does not support OPEN4_RESULT_PRESER=
VE_UNLINKED.
>
> The Linux NFSD does not guarantee that opened-but-deleted files were
> kept over reboots.
One final note...
Given the above, I suspect that the PUTFH, GETATTR after the REMOVE could
get a NFS4ERR_STALE if a server reboot were to happen right after the
REMOVE.

Thanks for all the comments, rick

>
> -Dai
>
> >
> > -Dai
> >> rick
> >>
> >>>> However if the "fail with NFS4ERR_STALE is supposed for happen after
> >>>> REMOVE for same client" then that is not what I am seeing.
> >>>> If you are curious, the packet trace is here. (Look at packet#58).
> >>>> https://urldefense.com/v3/__https://people.freebsd.org/*rmacklem/lin=
ux-remove.pcap__;fg!!ACWV5N9M2RV99hQ!IEcffaAAeLhuzaJUO5rQOv0jUUk4ltuMpfqT83=
lLFkRL9cqOZEvZ-8GGjvoqlVAQKi_FAAhsKEl5NjvS0OLJ$
> >>>>
> >>>> Btw, in case you are curious why I am doing this testing, I am tryin=
g
> >>>> to figure out a good way for the FreeBSD client to handle temporary
> >>>> files. Typically on POSIX they are done via the syscalls:
> >>>>
> >>>> fd =3D open("foo", O_CREATE ...);
> >>>> unlink("foo");
> >>>> write(fd,..), write(fd,..)...
> >>>> read(fd,...), read(fd,...)...
> >>>> close(fd);
> >>>>
> >>>> If this happens quickly and is not too much writing, the writes
> >>>> copy data into buffers/pages, the reads read the data out of
> >>>> the pages and then it all gets deleted.
> >>>>
> >>>> Unfortunately, the CB_RECALL forces the NFSv4.n client
> >>>> to do WRITE, WRITE,..COMMIT and then DELEGRETURN.
> >>>> Then the REMOVE throws all the data away on the NFSv4.n
> >>>> server.
> >>>> --> As such, I really like not doing the CB_RECALL for "same client"=
.
> >>>> My concern is "what happens to the delegation after the file object =
("foo")
> >>>> gets deleted?
> >>>> It either needs to be thrown away by the NFSv4.n server or the
> >>>> PUTFH, DELEGRETURN needs to work after the REMOVE.
> >>> The PUTFH and DELEGRETURN continue to work after the REMOVE. The open
> >>> stateid and delegation stateid on the server are destroyed only after
> >>> the client sends the DELEGRETURN and CLOSE.
> >>>
> >>>> Otherwise, the NFSv4.n server may get constipated by the delegations=
,
> >>>> which might be called stale, since the file object has been deleted.
> >>>>
> >>>> --> I can do PUTFH, GETATTR after REMOVE in the same compound,
> >>>>        to find out if the file object has been deleted. But then, if=
 a
> >>>>        PUTFH, DELEGRETURN fails with NFS4ERR_STALE, can I get
> >>>>        away with saying "the server should just discard the delegati=
on as
> >>>>        the client already has done so??.
> >>> You can try your test but I believe the PUTFH and GETATTR won't fail
> >>> after the REMOVE.
> >>>
> >>> -Dai
> >>>
> >>>> Thanks for your comments, rick
> >>>>
> >>>>> -Dai
> >>>>>
> >>>>>> (I don't think it is, but there is a discussion in 18.25.4 which s=
ays
> >>>>>> "When the determination above cannot be made definitively because
> >>>>>> delegations are being held, they MUST be recalled.." but everythin=
g
> >>>>>> above that is a may/MAY, so it is not obvious to me if a server re=
ally
> >>>>>> needs to case?)
> >>>>>>
> >>>>>> Any comments? Thanks, rick
> >>>>>> ps: I am amazed when I learn these things about NFSv4.n after all
> >>>>>>          these years.
> >>>>>>
> >
> > _______________________________________________
> > nfsv4 mailing list -- [email protected]
> > To unsubscribe send an email to [email protected]
>
> _______________________________________________
> nfsv4 mailing list -- [email protected]
> To unsubscribe send an email to [email protected]