[PATCH 4/4] NFSD: Send a meaningful CB_RECALL_ANY keep count

deleg_reaper() sets craa_objects_to_keep to zero on every
CB_RECALL_ANY. Per RFC 8881 Section 20.6.3, that asks the client to
retain no read or write delegation at all, whether or not the
delegation backs an open file.

The field names a count the client may keep. The client picks which
objects to return, because the server cannot read lack of recent
use as lack of usefulness. Zero leaves nothing to choose among. A
client that complies returns the delegations backing its open files
and reopens each one with CLAIM_DELEGATE_CUR, so NFSD trades a
delegation for an open stateid and recovers nothing. A client that
reads the zero as "unspecified" does nothing instead, and
nfsd4_cb_recall_any_done() inspects only the reply status, so NFSD
cannot tell the two apart.

Derive the keep count from cl_deleg_count and ask each client for a
single delegation. A larger request reaches delegations an
application still has open, and both callers re-arm while their
condition lasts. Skip a client holding one delegation rather than
send the zero again. That gate subsumes the list_empty() test above
it, and it belongs above the NFSD4_CALLBACK_RUNNING test_and_set. A
continue below that point latches the bit with no callback in
flight to clear it.

The Linux client ignores craa_objs_to_keep and returns unused
delegations from the type mask alone, so the count changes nothing
for it. The gate does. The reaper goes quiet for a client walked
down to one delegation.

Fixes: 44df6f439a17 ("NFSD: add delegation reaper to react to low memory condition")
Signed-off-by: Chuck Lever <[email protected]>
---
 fs/nfsd/nfs4state.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 3017a93261ff..e825ecb2ddfc 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -7944,6 +7944,7 @@ deleg_reaper(struct nfsd_net *nn)
 {
 	struct list_head *pos, *next;
 	struct nfs4_client *clp;
+	unsigned int count;
 
 	spin_lock(&nn->client_lock);
 	list_for_each_safe(pos, next, &nn->client_lru) {
@@ -7953,21 +7954,34 @@ deleg_reaper(struct nfsd_net *nn)
 			continue;
 		if (clp->cl_state != NFSD4_ACTIVE)
 			continue;
-		if (list_empty(&clp->cl_delegations))
-			continue;
 		if (atomic_read(&clp->cl_delegs_in_recall))
 			continue;
 		if (ktime_get_boottime_seconds() - clp->cl_ra_time < 5)
 			continue;
 		if (clp->cl_cb_state != NFSD4_CB_UP)
 			continue;
+		/*
+		 * This read races with hash_delegation_locked() and
+		 * unhash_delegation_locked() on other CPUs. A stale
+		 * value only defers this client to the next
+		 * laundromat pass.
+		 */
+		count = data_race(READ_ONCE(clp->cl_deleg_count));
+		if (count < 2)
+			continue;
 		if (test_and_set_bit(NFSD4_CALLBACK_RUNNING, &clp->cl_ra->ra_cb.cb_flags))
 			continue;
 
 		/* release in nfsd4_cb_recall_any_release */
 		kref_get(&clp->cl_nfsdfs.cl_ref);
 		clp->cl_ra_time = ktime_get_boottime_seconds();
-		clp->cl_ra->ra_keep = 0;
+		/*
+		 * Ask for one delegation at a time. A larger request
+		 * reaches delegations backing files that applications
+		 * still have open. Returning one of those trades a DELEG
+		 * stateid for an OPEN stateid and frees nothing.
+		 */
+		clp->cl_ra->ra_keep = count - 1;
 		clp->cl_ra->ra_bmval[0] = BIT(RCA4_TYPE_MASK_RDATA_DLG) |
 						BIT(RCA4_TYPE_MASK_WDATA_DLG) |
 						BIT(RCA4_TYPE_MASK_DIR_DLG);

-- 
2.54.0