[Bug 297019] [iscsi] initiator panics (vm_fault_lookup: fault on nofault entry) on retransmit after abrupt session reconnect during in-flight write I/O

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297019

            Bug ID: 297019
           Summary: [iscsi] initiator panics (vm_fault_lookup: fault on
                    nofault entry) on retransmit after abrupt session
                    reconnect during in-flight write I/O
           Product: Base System
           Version: Unspecified
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Many People
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

Panic seen on 14.4-RELEASE-p6, and the same code path confirmed present on
15.1-RELEASE by direct source comparison.
The root cause seems not release-specific.

= Observed panic (14.4-RELEASE-p6) =

panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe054b578000
cpuid = 10
time = 1784325462

KDB: stack backtrace:
#0 kdb_backtrace+0x5d
#1 vpanic+0x161
#2 panic+0x43
#3 vm_fault+0x1693
#4 vm_fault_trap+0x81
#5 trap_pfault+0x209
#6 calltrap+0x8
#7 in_cksum_skip_partial+0x24
#8 m_apply+0x83
#9 in_cksum_skip+0x2f
#10 in_delayed_cksum_o+0x4a
#11 ip_output+0x11a6
#12 tcp_default_output+0x1f48
#13 tcp_timer_rexmt+0x570
#14 tcp_timer_enter+0x107
#15 softclock_call_cc+0x126
#16 softclock_thread+0xe5
#17 fork_exit+0x81

Context: this box has active initiator sessions to a remote ctld target on
another host. The remote host went down (crashes or even clean shutdown, not
always the same) shortly this box logged hundreds of "no ping reply (NOP-In)
after N seconds; reconnecting" lines from the initiator's ping-timeout
mechanism right before the panic. Write I/O to that remote target was active at
the time.

= Possible root cause according to my very simple understanding of the code =

1. The initiator queues write data (both immediate data with the SCSI command,
and R2T-driven Data-Out) via icl_pdu_append_data_csio(), which passes
ICL_NOCOPY unconditionally for CAM_DIR_OUT transfers (sys/dev/iscsi/iscsi.c,
both call sites gated on CAM_DIR_OUT). This routes through
icl_soft_conn_pdu_append_bio() / icl_soft_conn_pdu_append_data(), building
zero-copy mbufs that reference the write's backing pages directly
(M_EXTPG/mb_alloc_ext_pgs for the bio path, m_extaddref() for the plain-buffer
path).

2. tcp_output.c's tcp_m_copym() (used for both original transmission and
retransmission) shares this reference via mb_dupcl() rather than copying bytes.
So a segment can still be retransmitted, referencing the original pages, well
after the PDU was first sent.

3. On ping-timeout, iscsi_maintenance_thread_reconnect() performs an abortive
close (SO_LINGER, l_linger=0 -- the comment in-source says this is intentional,
to discard outstanding data and RST immediately). iscsi_session_cleanup() ->
iscsi_session_terminate_tasks() -> iscsi_session_terminate_task() then calls
xpt_done() on every outstanding task UNCONDITIONALLY -- with no check on
whether the network stack has actually released its own reference to that
task's write data. This hands the backing buffer back to its owner (whatever
issued the raw-device write) while a queued-or-in-flight retransmit may still
reference it.

4. Notably, CTL's target-side equivalent, cfiscsi_session_terminate_tasks() in
ctl_frontend_iscsi.c, waits for exactly this class of condition
(cs_outstanding_ctl_pdus + a tsleep() poll loop) before proceeding. The
initiator side has no analogous wait. This asymmetry, within the same
subsystem, is what led me to this specific gap.

5. m_apply()'s EXTPG branch (m_apply_extpg_one(), sys/kern/uipc_mbuf.c)
dereferences PHYS_TO_DMAP(m->m_epg_pa[i] + pgoff + off) -- if that stored
physical address is no longer valid, this produces exactly a DMAP-region fault
inside a "nofault" vm_map_entry, matching the panic string and address pattern
observed.

I want to be explicit that step "the page actually becomes invalid, not just
logically stale" is the one link I could not be sure about as I don't have a
coredump from this box confirming the faulting mbuf's exact state. Everything
upstream of that (the unconditional CCB completion, the shared retransmit
reference, the asymmetry with the target-side code) is directly verified
against source, cited above.

= An idea for a potential fix =

Add a per-task network-side reference count (for example io_datamove_refs on
struct iscsi_outstanding), incremented when a write-carrying PDU is queued and
decremented only when icl_soft genuinely releases its own mbuf reference
(mirroring the existing
lock-free design of cfiscsi_pdu_done() in ctl_frontend_iscsi.c, which is the
existing, in-tree analog for this exact class of callback).
iscsi_session_terminate_task() waits for this to reach zero before completing
the CCB.

I'm posting this as an idea for a starting point for discussion, not a fix
Happy to iterate, or for someone more familiar with this code to take it in a
completely different direction if the analysis above is wrong somewhere.

-- 
You are receiving this mail because:
You are the assignee for the bug.