Support for polled IO completions for md-raid devices
Anton Gavriliuk <[email protected]>
| Newsgroups | gmane.linux.raid |
|---|---|
| Message-ID | <CAAiJnjqo0a0gHs3vMBn7kyAAm8LN95=3HPYdzpiSRe3fXtAMkg@mail.gmail.com> |
Hello
It is not surprising that polled IO completions significantly reduce
latency and increase IOPS. Here is an example with a single CM7-V
PCIe 5.0 NVMe SSD.
Interrupts,
[root@memverge4 ~]# fio --name=local_test --ioengine=io_uring
--rw=randread --bs=4K --numjobs=1 --direct=1 --filename=/dev/nvme1n1
--iodepth=64 --fixedbufs=1 --time_based=1 --runtime=20
local_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
(T) 4096B-4096B, ioengine=io_uring, iodepth=64
fio-3.41-49-gde3d
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=2936MiB/s][r=752k IOPS][eta 00m:00s]
local_test: (groupid=0, jobs=1): err= 0: pid=96203: Mon Dec 1 10:29:47 2025
read: IOPS=746k, BW=2915MiB/s (3056MB/s)(56.9GiB/20001msec)
slat (nsec): min=510, max=931130, avg=993.69, stdev=462.86
clat (usec): min=44, max=1056, avg=84.48, stdev=12.57
lat (usec): min=45, max=1057, avg=85.48, stdev=12.60
Polling,
[root@memverge4 ~]# fio --name=local_test --ioengine=io_uring
--rw=randread --bs=4K --numjobs=1 --direct=1 --filename=/dev/nvme1n1
--iodepth=64 --fixedbufs=1 --hipri=1 --time_based=1 --runtime=20
local_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
(T) 4096B-4096B, ioengine=io_uring, iodepth=64
fio-3.41-49-gde3d
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=4618MiB/s][r=1182k IOPS][eta 00m:00s]
local_test: (groupid=0, jobs=1): err= 0: pid=96543: Mon Dec 1 10:32:41 2025
read: IOPS=1174k, BW=4588MiB/s (4810MB/s)(89.6GiB/20001msec)
slat (nsec): min=510, max=909786, avg=663.38, stdev=410.86
clat (usec): min=34, max=1024, avg=53.69, stdev= 6.33
lat (usec): min=35, max=1024, avg=54.36, stdev= 6.34
Dut it doesn't work for md-raid device,
[root@memverge4 ~]# cat /proc/mdstat
Personalities : [raid0]
md127 : active raid0 nvme6n1[5] nvme0n1[3] nvme5n1[0] nvme2n1[1]
nvme3n1[4] nvme1n1[2]
18752907264 blocks super 1.2 512k chunks
unused devices: <none>
[root@memverge4 ~]#
[root@memverge4 ~]# fio --name=local_test --ioengine=io_uring
--rw=randread --bs=4K --numjobs=1 --direct=1 --filename=/dev/md127
--iodepth=64 --fixedbufs=1 --hipri=1 --time_based=1 --runtime=20
local_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
(T) 4096B-4096B, ioengine=io_uring, iodepth=64
fio-3.41-49-gde3d
Starting 1 process
fio: file /dev/md127 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to
get rid of this warning.
fio: io_u error on file /dev/md127: Operation not supported: read
offset=18761783697408, buflen=4096
fio: io_u error on file /dev/md127: Operation not supported: read
offset=17577830526976, buflen=4096
fio: io_u error on file /dev/md127: Operation not supported: read
offset=16065491521536, buflen=4096
fio: io_u error on file /dev/md127: Operation not supported: read
offset=15054980657152, buflen=4096
...............................................
I'm not a developer or programmer.
May I ask as a feature request to add support for polling IO
completions for md-raid devices ?
Anton