Re: [PATCH V2] nvme: Add module reference counting for multipath devices

On 2026-08-22 19:46, Sagi Grimberg wrote:

> OK, I now read the original report regarding nvme root device. please 
> disregard.
Hi Sagi,

Thanks for taking the time to review this!

If multipath NVMe is the root device and there is no single NVMe device 
in the system, the tester can still rmmod nvme because the NVMe module's 
reference count is 0 (PCIe).

For example,
System has 2 multipath nvme devices(nvme0 has 1 NS and nvme1 has 2 NS).
# nvme list
Node                  Generic               SN                   Model   
                                  Namespace  Usage                      
Format           FW Rev
--------------------- --------------------- -------------------- 
---------------------------------------- ---------- 
-------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            S6RUNE0R900042       1.6TB 
NVMe Gen4 U.2 SSD III              0x1        400.08  GB / 400.08  GB    
   4 KiB +  0 B   REV.SN66
/dev/nvme1n1          /dev/ng1n1            S6KZNE0RA00056       800GB 
NVMe Gen4 U.2 SSD                  0x1        400.08  GB / 400.08  GB    
   4 KiB +  0 B   REV.SN4B
/dev/nvme1n2          /dev/ng1n2            S6KZNE0RA00056       800GB 
NVMe Gen4 U.2 SSD

Without patch, reference count of nvme module is 0, "rmmod nvme" causes 
system crashing.
#lsmod|grep nvme
  nvme_tcp              262144  0
nvme                  262144  0
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

static int nvme_ns_open(struct nvme_ns *ns)
{

         /* should never be called due to GENHD_FL_HIDDEN */
         if (WARN_ON_ONCE(nvme_ns_head_multipath(ns->head))) ---------> 
skip multipath devs
                 goto fail;
         if (!nvme_get_ns(ns))
                 goto fail;
         if (!try_module_get(ns->ctrl->ops->module))
                 goto fail_put_ns;

}

#rmmod nvme
[  434.859171] block nvme2n1: no available path - failing I/O
[  434.859204] XFS (nvme2n1p2): log I/O error -5
[  434.859211] XFS (nvme2n1p2): Filesystem has been shut down due to log 
error (0x2).
[  434.859215] XFS (nvme2n1p2): Please unmount the filesystem and 
rectify the problem(s).
[root@ltcrain119-lp4 nvme_upstream]# [  439.349490] XFS (dm-0): metadata 
I/O error in "xfs_imap_to_bp+0x74/0x108 [xfs]" at daddr 0x73d4ca0 len 32 
error 5
[  439.349651] XFS (dm-0): metadata I/O error in 
"xfs_imap_to_bp+0x74/0x108 [xfs]" at daddr 0x73d4ca0 len 32 error 5
[  439.353244] coredump: 26799(fwupd): 
|/usr/lib/systemd/systemd-coredump pipe failed
[  457.438961] XFS (dm-0): log I/O error -5
[  457.438998] XFS (dm-0): Filesystem has been shut down due to log 
error (0x2).
[  457.439005] XFS (dm-0): Please unmount the filesystem and rectify the 
problem(s).

With patch, reference count of nvme is 6 after system boots up.
2 paths for nvme0 and 4 paths for nvme1:
Can't do "rmmod nvme" command now.

#lsmod|grep nvme
# lsmod|grep nvme
nvme_tcp              262144  0
nvme                  262144  6
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

# rmmod nvme
rmmod: ERROR: Module nvme is in use

If I remove 1 NS nvme device, reference count decreases 6 – 2 = 4 paths 
in total.
# lsmod|grep nvme
nvme_tcp              262144  0
nvme                  262144  4
nvme_fabrics          262144  1 nvme_tcp
nvme_core             458752  7 nvme_tcp,nvme,nvme_fabrics
nvme_keyring          262144  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth             262144  1 nvme_core

If you need any additional information or have any other considerations, 
please let me know.
Could you please help review the patch and share any feedback?

Thanks,
Wen