[PATCH 12/21] NFSv4/flexfiles: Honor ndc_immediate on CB_NOTIFY_DEVICEID CHANGE

When a CHANGE notification carries ndc_immediate, RFC 8881 Section
20.12 says the change is enforced immediately and the client might not
be able to complete pending I/O.  In addition to un-pinning the stripe's
device node, mark the old node unavailable so paths still holding a
reference stop selecting it; I/O the server rejects is re-driven onto
the freshly resolved mapping by the existing DS error handling.

The walk can also exchange out a node that already carries the new
mapping: a re-resolve that completed between the unhash and the walk
reaching the stripe, raced by this walk or by the walk of a later
CHANGE for the same deviceid.  Ripping such a node out is harmless (it
is still hashed, so the next I/O re-pins it from the cache), but it
must not be marked unavailable.  Stale vintages are distinguishable by
hashed-ness: every superseded node was unhashed before its walk began
and is never re-inserted, while a fresh node is inserted before it is
installed.  Only mark nodes that are no longer hashed.

Without ndc_immediate, pending I/O drains on the old mapping and only
new I/O re-resolves, as before.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Benjamin Coddington <bcodding-F/[email protected]>
---
 fs/nfs/flexfilelayout/flexfilelayout.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index bb27bbea6c95..f042551dd23a 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -2535,6 +2535,23 @@ static void ff_layout_reresolve_deviceid(struct pnfs_layout_hdr *lo,
 				xchg(&mirror->dss[dss_id].mirror_ds, NULL));
 			if (IS_ERR_OR_NULL(old))
 				continue;
+			/*
+			 * ndc_immediate: the server is enforcing the change
+			 * now and pending I/O may not complete.  Mark the
+			 * old node unavailable so anyone still holding a
+			 * reference stops selecting it; I/O the server
+			 * rejects is re-driven onto the new mapping by the
+			 * existing DS error handling.
+			 *
+			 * Only stale vintages are unhashed: a node still
+			 * hashed was fetched after the stale cache entry
+			 * was removed and already carries the new mapping
+			 * (it can be exchanged out here by a walk that
+			 * lost a race with the re-resolve, or with the
+			 * walk of a later CHANGE); don't mark it.
+			 */
+			if (immediate && hlist_unhashed(&old->id_node.node))
+				nfs4_mark_deviceid_unavailable(&old->id_node);
 			list_add(&old->id_node.put_list, head);
 		}
 	}
-- 
2.53.0