[Bug 297252] NFSv4.1 client permanently wedges after NFS4ERR_BADSESSION when the live MDS session is left defunct with no replacement

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D297252

            Bug ID: 297252
           Summary: NFSv4.1 client permanently wedges after
                    NFS4ERR_BADSESSION when the live MDS session is left
                    defunct with no replacement
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

Created attachment 273424
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D273424&action=
=3Dedit
patch

Solving AWS EFS mount problem with my EC2 instances, with the help of claude
code.

An NFSv4.1 mount can wedge permanently: every process touching it blocks in
uninterruptible D state (WCHAN "nfsbadse") and only a reboot recovers.
It is triggered when the server returns NFS4ERR_BADSESSION on SEQUENCE while
the
ClientID is also stale (NFS4ERR_STALE_CLIENTID).
Observed against Amazon EFS, and reproduced without EFS against a plain loc=
al
nfsd (see below).

Root cause

Two guards that each prevent a recovery storm together form a trap with no
exit:

1. sys/fs/nfs/nfs_commonkrpc.c, BADSESSION handler in newnfs_request():
recovery
   is only (re)initiated when the live MDS session has nfsess_defunct =3D=
=3D 0.
Once
   the live session is already defunct, every further
SEQUENCE->NFS4ERR_BADSESSION
   is silently swallowed and recovery is never triggered again.

2. sys/fs/nfs/nfs_commonsubs.c, nfsv4_sequencelookup(): returns
NFS4ERR_BADSESSION
   for any op on a defunct session, so foreground ops spin the 1s "Badsessi=
on
   looping" retry forever.


How the live session is left defunct with no replacement:

The recover_done_time gate in nfscl_renewthread()
(sys/fs/nfsclient/nfs_clstate.c) allows only one full recovery per nfsc_ren=
ew
(lease/2). A second BADSESSION within that window forces nfscl_recover() to
call nfsrpc_setclient() with retok=3Dtrue, i.e. CreateSession with the exta=
nt
ClientID only.
Against a stale ClientID that CreateSession fails, nfsrpc_setclient() retur=
ns
NFSERR_IO (sys/fs/nfsclient/nfs_clrpcops.c), which nfscl_recover() does not
retry. But the BADSESSION handler had already marked the live session defun=
ct,
so it stays at the head of nm_sess with no non-defunct
replacement, and guard 1 then ensures recovery is never attempted again.

Evidence (live EFS capture, vfs.nfs.debuglevel=3D1):
Recovery fires and succeeds 4x via the ExchangeID+CreateSession fallback, t=
hen
after two recoveries 45s apart the live MDS session is defunct with no
replacement; from there the renew thread hits SEQUENCE->NFS4ERR_BADSESSION
every ~45s for 2h17m with zero "Initiate recovery", then foreground access
spins "Badsession looping" at 1/s until reboot. nfsstat -c freezes
(ExchangeId/CreateSess pinned).

Reproduce without EFS (plain local nfsd + a fault-injecting TCP proxy that
returns
NFS4ERR_BADSESSION on SEQUENCE and NFS4ERR_STALE_CLIENTID on CREATE_SESSION=
):
  sysctl vfs.nfsd.server_max_minorversion4=3D2 vfs.nfsd.nfs_privport=3D0
  sysctl vfs.nfs.debuglevel=3D1
  # export /export over "V4: /export"; run proxy on :2050 -> :2049, arm 4s/=
15s
  mount_nfs -o nfsv4,minorversion=3D1,hard,retrans=3D2,port=3D2050 127.0.0.=
1:/ /mnt/t
  # drive concurrent ls/stat on /mnt/t in a loop
The unpatched client wedges in ~3 min (frozen nfsstat, "Badsession looping"=
 at
1/s with no "Initiate recovery", D-state "nfsbadse").

Fix:
In the BADSESSION handler in newnfs_request(), drop the nfsess_defunct =3D=
=3D 0
term from the re-trigger condition.
The NFSCLFLAGS_RECVRINPROG | NFSCLFLAGS_RECOVER check already present is wh=
at
prevents a recovery storm; the nfsess_defunct =3D=3D 0 test is redundant fo=
r that
purpose and is what makes the wedge permanent. With it removed, a BADSESSIO=
N on
an
already-defunct live session re-arms NFSCLFLAGS_RECOVER. Setting nfsess_def=
unct
=3D 1 is idempotent.
This self-heals: CreateSession-only retries repeat one BADSESSION apart unt=
il
the nfsc_renew window rolls over, after which a full ExchangeID+CreateSessi=
on
recovery succeeds.

--=20
You are receiving this mail because:
You are the assignee for the bug.=