Re: [PATCH 00/21] NFS: flexfiles device notifications and caching for wide striped layouts

Note to reviewers - this one had numerous problems identified by sashiko:
https://sashiko.dev/#/patchset/cover.1786653063.git.bcodding%40hammerspace.com

I'm working on a v2..

Ben

On 13 Aug 2026, at 16:42, Benjamin Coddington wrote:

> A flexfiles layout striped across many data servers puts hundreds to
> ~1000 distinct deviceIDs in play for a single mount.  This series makes
> the client's CB_NOTIFY_DEVICEID handling actually work under live
> layouts, implements the deviceID-deletion race recovery RFC 8881 asks
> for, and scales the device caches to that number of devices.
>
> Patches 1-4 are standalone fixes (Cc: stable) for latent bugs in the
> striped-layout paths.  The CB_NOTIFY_DEVICEID decoder gated consuming
> ndc_immediate on the layout type instead of the notification type, so
> the flag was silently lost and any multi-item cnda_changes<> array
> misaligned after the first entry.  Read data-server selection truncated
> the file offset to 32 bits, picking the wrong stripe's DS for
> availability probing and coalescing above 4 GiB.  And page coalescing
> was bounded on a segment-relative offset in both the flexfiles and files
> layouts, so an unaligned segment let a coalesced I/O straddle a stripe
> boundary and send the bytes past it to the wrong data server.
>
> Patches 5-13 make CHANGE notifications work under live layouts.  RFC 8881
> Section 12.2.10 has CB_NOTIFY_DEVICEID CHANGE exist precisely so a server
> can re-point a deviceID without recalling the layouts that reference it,
> but the client's only response today is to unhash the cached device --
> which never reaches the references pinned inside the layout driver's
> segments, so I/O keeps going to the old mapping until the layouts are
> freed.  Each in-flight I/O is given its own device-node reference, the
> mirror's pinned pointer becomes RCU-managed, and a new layout-driver
> hook re-resolves the device in place, honoring ndc_immediate.  A
> GETDEVICEINFO reply that raced a CHANGE is discarded rather than
> installed stale.
>
> Patches 14-17 implement the deviceID-deletion race of RFC 8881 Section
> 18.40.4: layouts still referencing a deleted deviceID are tested with
> TEST_STATEID, revoked state is recovered, and the deletion is confirmed
> with GETDEVICEINFO before escalating to lease recovery.  A DELETE that
> no live layout references keeps today's cheap path.
>
> Patches 18-21 scale the device caches.  The global deviceid hash grows
> from 32 to 256 buckets (a load factor of ~31 at 1000 devices today), and
> the per-net data-server cache moves from a single list to hash buckets
> keyed by the DS address set -- which also fixes a latent aliasing bug,
> since the old comparator matched on subset rather than equality and
> merged two data servers whenever one's address set contained the other's.
> Last, the flexfiles driver gains a dataserver_nconnect module parameter
> alongside its existing dataserver_timeo and dataserver_retrans knobs:
> data-server clients otherwise inherit the MDS nconnect, which at ~1000
> DSes and nconnect=16 means ~16k sockets and their slot tables for little
> gain on a workload striping across the DSes anyway.  It defaults to 0,
> preserving today's inherit-from-MDS behavior.
>
> Tested end-to-end against reffs, a notification-capable reference
> server, at 256 and 1000 devices, including KASAN and lockdep runs under
> concurrent notification storms.
>
> A related series, "NFS: size the LAYOUTGET reply buffer for wide
> flexfiles layouts", lifts the single-page LAYOUTGET reply buffer that
> today caps a segment at roughly 28 stripes.  The two are independent and
> apply cleanly in either order.
>
> Benjamin Coddington (21):
>   pNFS: Fix CB_NOTIFY_DEVICEID CHANGE to consume ndc_immediate
>   NFSv4/flexfiles: Use the full 64-bit offset for read DS selection
>   NFSv4/flexfiles: Bound page coalescing on the absolute stripe offset
>   NFSv4/filelayout: Anchor page coalescing on pattern_offset
>   NFSv4/flexfiles: Reference the device node across DS setup
>   NFSv4/flexfiles: Carry the device node reference across each I/O
>   NFSv4/flexfiles: Hold a device node reference for layoutstats encoding
>   NFSv4/flexfiles: Make the pinned device node pointer RCU-managed
>   pNFS: Add a reresolve_deviceid layout driver hook
>   NFSv4/flexfiles: Implement in-place device re-resolve on CHANGE
>   NFSv4: Dispatch CB_NOTIFY_DEVICEID CHANGE to an in-place refresh
>   NFSv4/flexfiles: Honor ndc_immediate on CB_NOTIFY_DEVICEID CHANGE
>   pNFS: Discard a GETDEVICEINFO reply that raced a CHANGE notification
>   pNFS: Add deviceid reference query and collection walkers
>   NFSv4/pnfs: Recover revoked layouts on a deleted deviceID
>   NFSv4/pnfs: Confirm a deviceID delete via GETDEVICEINFO
>   NFSv4/pnfs: Dispatch CB_NOTIFY_DEVICEID DELETE to race recovery
>   NFSv4/pnfs: Grow the deviceid cache hash table
>   NFSv4/pnfs: Re-home the data-server cache onto hash buckets
>   NFSv4/pnfs: Key the data-server cache by its address set
>   NFSv4/flexfiles: Add a dataserver_nconnect cap
>
>  fs/nfs/callback_proc.c                    |  30 +-
>  fs/nfs/callback_xdr.c                     |   2 +-
>  fs/nfs/client.c                           |   6 +-
>  fs/nfs/filelayout/filelayout.c            |   8 +-
>  fs/nfs/filelayout/filelayoutdev.c         |   2 +-
>  fs/nfs/flexfilelayout/flexfilelayout.c    | 383 ++++++++++++++--------
>  fs/nfs/flexfilelayout/flexfilelayout.h    |  44 +--
>  fs/nfs/flexfilelayout/flexfilelayoutdev.c | 193 +++++++----
>  fs/nfs/internal.h                         |   3 +-
>  fs/nfs/netns.h                            |   5 +-
>  fs/nfs/nfs3client.c                       |   9 +-
>  fs/nfs/nfs4_fs.h                          |   2 +
>  fs/nfs/nfs4client.c                       |   7 +-
>  fs/nfs/nfs4proc.c                         | 113 +++++++
>  fs/nfs/nfs4state.c                        |   3 +
>  fs/nfs/pnfs.c                             | 292 +++++++++++++++++
>  fs/nfs/pnfs.h                             |  81 ++++-
>  fs/nfs/pnfs_dev.c                         |  35 +-
>  fs/nfs/pnfs_nfs.c                         |  93 +++++-
>  include/linux/nfs_fs_sb.h                 |   2 +
>  include/linux/nfs_xdr.h                   |   2 +
>  21 files changed, 1069 insertions(+), 246 deletions(-)
>
> base-commit: db2ddb87143519e20a95aa36c60b36107b736a58
> -- 
> 2.53.0