Nvme-rdma: Long IO HANG (~4.8h) during link failure with dm-multipath (Kernel 6.6)

Hi all,
We encountered a severe I/O hang issue during NVMe-oF (RoCEv2) link failure=
 when native multipath is disabled and dm-multipath is used instead.
Based on our test results, this issue is highly likely located within the r=
econnect/backoff mechanism of the vendor's out-of-tree driver (NVIDIA DOCA-=
Host 3.3.0). The attachment contains the logs captured when the fault occur=
s.
We look forward to the vendor analyzing this driver-side issue, but we woul=
d also like to seek advice from community experts regarding potential optim=
ization suggestions for the block layer and NVMe-oF (NOF) collaboration mec=
hanisms to guard against such behavior.

Environment
*	Kernel: 6.6.0 (openEuler 24.03 LTS SP4 baseline)
*	Hardware: Intel Xeon Gold 5220R / Mellanox ConnectX-5 (MT27800, FW: 26.01=
-1.0.0)
*	Storage: Huawei OceanStor Dorado (NVMe over RoCEv2)
*	Configuration: nvme_core.multipath=3DN, dm-multipath enabled.
*	Fabric Parameters: reconnect_delay =3D 10, ctrl_loss_tmo =3D 600 (or ctrl=
_loss_tmo =3D 10 for testing)

Problem Description & Observations
When we manually inject a link fault on one of the paths and restart multip=
athd, the multipathd process gets stuck in the D state (the path detection =
I/O does not return). Application I/O hangs for a very long time due to a r=
equeue ping-pong loop.
1.	The Requeue Ping-Pong: After the link goes down, the NVMe controller tra=
nsitions to the NVME_CTRL_CONNECTING state. During this time, nvme_fail_non=
ready_command() returns BLK_STS_RESOURCE. Upon receiving this, the block la=
yer (blk-mq) immediately requeues the request. This causes the I/O to endle=
ssly "ping-pong" between the block layer and nvme-core.
2.	Abnormal Retry Interval: Based on ctrl_loss_tmo / reconnect_delay =3D 60=
0 / 10, the subsystem is expected to retry 60 times. However, the actual me=
asured interval between two consecutive retries is stretched to 288s - 290s=
, completely ignoring the configured reconnect_delay =3D 10. The total hang=
 time reaches 17,340 seconds (~4.8 hours).
3.	Single Retry Test: To isolate the issue, we set ctrl_loss_tmo =3D 10 (wh=
ich triggers only 1 retry before tearing down the controller). Even in this=
 case, the I/O still HANGS for exactly ~290 seconds. This indicates that th=
e problem is not cumulative retry multiplication, but rather that the very =
first reconnect attempt or the single reconnect worker itself is being bloc=
ked/delayed internally for ~290 seconds by the underlying driver stack.

In-box Driver Contrast
If we switch back to the kernel in-box (mainline upstream) mlx5 driver, thi=
s issue DOES NOT occur. The NVMe controller status updates rapidly upon lin=
k failure, and failover finishes within seconds. This further confirms that=
 the 290-second blocking behavior is specific to the DOCA driver stack.

Technical Consultation
The most appropriate solution is for the vendor to fix this issue within th=
eir driver. However, if the vendor driver cannot technically resolve it, ar=
e there any potential optimization mechanisms within the block layer or the=
 multipath subsystem to mitigate this?
For example:
1.	Block Layer: During the NVME_CTRL_CONNECTING state, if requests continuo=
usly receive BLK_STS_RESOURCE, should the block layer introduce an exponent=
ial backoff mechanism or a maximum retry threshold? Blindly requeuing these=
 requests seems to create an infinite loop that leaves the system vulnerabl=
e to worker starvation or long hangs if the underlying driver blocks.
2.	Multipath Subsystem: The multipath subsystem's path detection mechanism =
for NVMe devices could be optimized. I believe that path-checking I/Os (suc=
h as those sent by multipathd to verify link sanity) should not be allowed =
to retry indefinitely under these conditions, as it completely stalls the f=
ailover process.

Any insights or architectural suggestions on how the block layer can better=
 handle or guard against such non-responsive driver behavior would be great=
ly appreciated.

Thanks.