Re: [nfsv4] simple NFSv4.1/4.2 test of remove while holding a delegation

On Thu, Jun 12, 2025 at 5:59=E2=80=AFAM David Noveck <[email protected]>=
 wrote:
>
>
>
> On Wed, Jun 11, 2025, 4:58=E2=80=AFPM Rick Macklem <[email protected]=
m> wrote:
>>
>> On Wed, Jun 11, 2025 at 9:28=E2=80=AFAM David Noveck <[email protected]=
om> wrote:
>> >
>> >
>> >
>> > On Mon, Jun 9, 2025, 7:35=E2=80=AFPM Rick Macklem <rick.macklem@gmail.=
com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I hope you don't mind a cross-post, but I thought both groups
>> >> might find this interesting...
>> >
>> >
>> > I find it interesting, but I can't speak.for either group.
>> >>
>> >>
>> >> I have been creating a compound RPC that does REMOVE and
>> >> then tries to determine if the file object has been removed and
>> >> I was surprised to see quite different results from the Linux knfsd
>> >> and Solaris 11.4 NFSv4.1/4.2 servers. I think both these servers
>> >> provide FH4_PERSISTENT file handles, although I suppose I
>> >> should check that?
>> >>
>> >> First, the test OPEN/CREATEs a regular file called "foo" (only one
>> >> hard link) and acquires a write delegation for it.
>> >> Then a compound does the following:
>> >> ...
>> >> REMOVE foo
>> >> PUTFH fh for foo
>> >> GETATTR
>> >>
>> >> For the Solaris 11.4 server, the server CB_RECALLs the
>> >> delegation and then replies NFS4ERR_STALE for the PUTFH above.
>> >> (The FreeBSD server currently does the same.)
>> >>
>> >> For a fairly recent Linux (6.12) knfsd, the above replies NFS_OK
>> >> with nlinks =3D=3D 0 in the GETATTR reply.
>> >>
>> >> Hmm. So I've looked in RFC8881 (I'm terrible at reading it so I
>> >> probably missed something) and I cannot find anything that states
>> >> either of the above behaviours is incorrect.
>> >> (NFS4ERR_STALE is listed as an error code for PUTFH, but the
>> >> description of PUTFH only says that it sets the CFH to the fh arg.
>> >> It does not say anything w.r.t. the fh arg. needing to be for a file
>> >> that still exists.) Neither of these servers sets
>> >> OPEN4_RESULT_PRESERVE_UNLINKED in the OPEN reply.
>> >>
>> >> So, it looks like "file object no longer exists" is indicated either
>> >> by a NFS4ERR_STALE reply to either PUTFH or GETATTR
>> >> OR
>> >> by a successful reply, but with nlinks =3D=3D 0 for the GETATTR reply=
.
>> >>
>> >> To be honest, I kinda like the Linux knfsd version, but I am wonderin=
g
>> >> if others think that both of these replies is correct?
>> >
>> >
>> > I think they are both correct.  It seems to me that an attempt to choo=
se one of these as preferred and deprecating the other should be rejected s=
ince it unjustiably imposes a particular design choice on the server.
>> >>
>> >>
>> >> Also, is the CB_RECALL needed when the delegation is held by
>> >> the same client as the one doing the REMOVE?
>> >
>> >
>> > I think so.
>> From a practical point of view, I am not convinced it is needed.
>
>
> I see your point. The problem is in clearly explaining when it is not nee=
ded.
>
>> The server can determine if the REMOVE actually deleted the
>> file and, if it did, can throw away any delegation record(s) for the
>> file object.
>> The client knows it has a delegation and can either DELEGRETURN
>> it or throw it away if it knows the file object has been deleted and the
>> associated file handle is no longer valid (it receives a NFS4ERR_STALE
>> from the server for it).
>
>
> That case can be dealt with by stating that delegations associated with i=
nvalid fh's can be considered effectively obliterated without being returne=
d.
>
> The troublesome case is for servers that do not invalidate the fh.  They =
are still able to do a DELEGRETURN, so it would have to be separately state=
d that if it is done after removal it MAY fail that and, if it does, the cl=
ient can forget it
>
> I don't want to get into a situation in which client have to check the va=
lidity of the fh after the remove.
Given that the NFSv4.1/4.2 server might not recall the delegation,
I think the client has options for handling the case where it holds
a delegation for the file object it is doing a REMOVE of.
Here's three possibilities I can think of:
1 - Do the write flush and DELEGRETURN before REMOVE.
     (This is what the three extant clients I have for testing do.)
2 - Do the write flush and DELEGRETURN asynchronously after
     REMOVE, accepting that the write flush or DELEGRETURN
     may fail with NFS4ERR_STALE. (Having it fail with NFS4ERR_STALE
     is fine and reduces overhead.)
3 - Do a PUTFH and GETATTR numlinks after the REMOVE.
     if either of these fail with NFS4ERR_STALE
          - The client can discard dirty buffers/pages and the delegation
     else if numlinks > 0
          - Do nothing. The delegation is still valid.
     else
          - Do #2.
So, only #3 tries to check after the remove, but is really just an
optimization of #2.

rick
>>
>>
>> Also, wearing my pragmatic practitioner's hat, since the Linux knfsd
>> does not do a CB_RECALL now and has shipped this way to who
>> knows how many users, declaring that it must be CB_RECALL'd
>> does not seem useful?
>
>
> I think MUST would be wrong.  The trouble is that SHOULD would also be wr=
ong.  Stay tuned.
>>
>>
>> rick
>>
>> >
>> >> (I don't think it is, but there is a discussion in 18.25.4 which says
>> >> "When the determination above cannot be made definitively because
>> >> delegations are being held, they MUST be recalled.." but everything
>> >> above that is a may/MAY, so it is not obvious to me if a server reall=
y
>> >> needs to case?)
>> >
>> >
>> > This should be more clear.  Will be looking at a possible change in th=
e next rfc5661bis draft.
>
>
> Those changes need to be acceptable to pragmatic practitioners such as yo=
u but should be clear enough to helpbthose who are not looking at an existi=
ng implementation.
>
>> >>
>> >>
>> >> Any comments? Thanks, rick
>> >> ps: I am amazed when I learn these things about NFSv4.n after all
>> >>       these years.
>> >>
>> >> _______________________________________________
>> >> nfsv4 mailing list -- [email protected]
>> >> To unsubscribe send an email to [email protected]