Re: --bitmap=lockless across server reboot
Anton Gavriliuk <[email protected]>
| Newsgroups | gmane.linux.raid |
|---|---|
| Message-ID | <CAAiJnjr_FcaivTUy+gJwo6ob7BLivw3eU0_huWMA7hOcw9raPg@mail.gmail.com> |
I remembered how to reproduce.
The raid is gone after I performed precondition for performance tests
and reboot.
After reboot there are no issues in dmesg and messages.
[root@qdevice ~]# fio --name=raid5_final --rw=write --bs=2048k
--filename=/dev/md0 --direct=1 --numjobs=1 --iodepth=64
--group_reporting --ioengine=io_uring --time_based --runtime=1800
raid5_final: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W)
2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=io_uring, iodepth=64
fio-3.42-29-gd569
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=8316MiB/s][w=4158 IOPS][eta 00m:00s]
raid5_final: (groupid=0, jobs=1): err= 0: pid=5070: Mon May 25 19:56:09 2026
write: IOPS=4088, BW=8176MiB/s (8574MB/s)(14.0TiB/1800013msec)
slat (usec): min=15, max=1287, avg=36.23, stdev=11.79
clat (usec): min=2072, max=43284, avg=15618.22, stdev=3312.23
lat (usec): min=2094, max=43313, avg=15654.45, stdev=3312.52
clat percentiles (usec):
| 1.00th=[ 8356], 5.00th=[10421], 10.00th=[11469], 20.00th=[12911],
| 30.00th=[13829], 40.00th=[14615], 50.00th=[15533], 60.00th=[16319],
| 70.00th=[17171], 80.00th=[18220], 90.00th=[19792], 95.00th=[21103],
| 99.00th=[23987], 99.50th=[25297], 99.90th=[28181], 99.95th=[29492],
| 99.99th=[32375]
bw ( MiB/s): min= 7736, max= 9132, per=100.00%, avg=8178.90,
stdev=166.52, samples=3599
iops : min= 3868, max= 4566, avg=4089.42, stdev=83.27, samples=3599
lat (msec) : 4=0.01%, 10=3.68%, 20=87.01%, 50=9.32%
cpu : usr=10.21%, sys=5.19%, ctx=5637748, majf=0, minf=754452
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,7358832,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0.00ns, window=0.00ns, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=8176MiB/s (8574MB/s), 8176MiB/s-8176MiB/s
(8574MB/s-8574MB/s), io=14.0TiB (15.4TB), run=1800013-1800013msec
Disk stats (read/write):
md0: ios=20/14716476, sectors=2560/30139342848, merge=0/0,
ticks=0/125528858, in_queue=125528858, util=100.00%,
aggrios=68/7358896, aggsectors=3046/7535444480, aggrmerge=0/0,
aggrticks=6/413853, aggrin_queue=413859, aggrutil=21.24%
nvme0n1: ios=48/7358896, sectors=1680/7535444480, merge=0/0,
ticks=6/408945, in_queue=408951, util=20.67%
nvme3n1: ios=67/7358896, sectors=3992/7535444480, merge=0/0,
ticks=4/433875, in_queue=433879, util=21.24%
nvme2n1: ios=118/7358896, sectors=2960/7535444480, merge=0/0,
ticks=7/405682, in_queue=405689, util=20.45%
nvme5n1: ios=75/7358896, sectors=5136/7535444480, merge=0/0,
ticks=9/407693, in_queue=407702, util=20.42%
nvme1n1: ios=51/7358896, sectors=2064/7535444480, merge=0/0,
ticks=7/409760, in_queue=409767, util=20.59%
nvme4n1: ios=54/7358896, sectors=2448/7535444480, merge=0/0,
ticks=5/417163, in_queue=417168, util=20.58%
[root@qdevice ~]#
[root@qdevice ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme6n1 259:1 0 447.1G 0 disk
├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
├─nvme6n1p2 259:3 0 1G 0 part /boot
├─nvme6n1p3 259:4 0 220G 0 part /
├─nvme6n1p4 259:5 0 1G 0 part
└─nvme6n1p5 259:6 0 220G 0 part
nvme0n1 259:9 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme1n1 259:10 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme4n1 259:15 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme2n1 259:16 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme5n1 259:17 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme3n1 259:18 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
[root@qdevice ~]#
[root@qdevice ~]# reboot
login as: root
[email protected]'s password:
Web console: https://qdevice:9090/ or https://10.72.14.186:9090/
Last login: Mon May 25 19:25:38 2026 from 10.72.12.178
[root@qdevice ~]# uptime
20:01:59 up 0 min, 1 user, load average: 0.87, 0.26, 0.09
[root@qdevice ~]#
[root@qdevice ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme5n1 259:1 0 447.1G 0 disk
├─nvme5n1p1 259:2 0 1G 0 part /boot/efi
├─nvme5n1p2 259:3 0 1G 0 part /boot
├─nvme5n1p3 259:4 0 220G 0 part /
├─nvme5n1p4 259:5 0 1G 0 part
└─nvme5n1p5 259:6 0 220G 0 part
nvme1n1 259:8 0 2.9T 0 disk
nvme3n1 259:10 0 2.9T 0 disk
nvme4n1 259:13 0 2.9T 0 disk
nvme2n1 259:14 0 2.9T 0 disk
nvme0n1 259:17 0 2.9T 0 disk
nvme6n1 259:18 0 2.9T 0 disk
[root@qdevice ~]#
[root@qdevice ~]# uptime
20:02:40 up 1 min, 1 user, load average: 0.46, 0.24, 0.09
[root@qdevice ~]#
Anton
пн, 25 мая 2026 г. в 18:08, Anton Gavriliuk <[email protected]>:
>
> Hhmm.... I just re-checked it with latest kernel (7.1-rc5) on Fedora
> Server 44, and now it works, log below. I don't remember on which
> kernel I saw that issue (7.1-rc3 or 7.1-rc4)
>
> BTW, earlier you mentioned that you are working on raid5 performance
> improvements,
>
> "“BTW we're working on performance
> improvement for large raid5 arrays for at most 64 disks, results looks great.
> However, it might take sometime before we push our work to upstream.”
>
> Are there any progress ?, I'm very interested to test on PCIe 5.0 NVMe drives.
>
> This is Fedora Server 44 (up to date) with compiled kernel 7.1-rc5
>
> [root@qdevice ~]# cat /etc/*release
> Fedora release 44 (Forty Four)
> NAME="Fedora Linux"
> VERSION="44 (Server Edition)"
> RELEASE_TYPE=stable
> ID=fedora
> VERSION_ID=44
> VERSION_CODENAME=""
> PRETTY_NAME="Fedora Linux 44 (Server Edition)"
> ANSI_COLOR="0;38;2;60;110;180"
> LOGO=fedora-logo-icon
> CPE_NAME="cpe:/o:fedoraproject:fedora:44"
> HOME_URL="https://fedoraproject.org/"
> DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f44/"
> SUPPORT_URL="https://ask.fedoraproject.org/"
> BUG_REPORT_URL="https://bugzilla.redhat.com/"
> REDHAT_BUGZILLA_PRODUCT="Fedora"
> REDHAT_BUGZILLA_PRODUCT_VERSION=44
> REDHAT_SUPPORT_PRODUCT="Fedora"
> REDHAT_SUPPORT_PRODUCT_VERSION=44
> SUPPORT_END=2027-05-19
> VARIANT="Server Edition"
> VARIANT_ID=server
> Fedora release 44 (Forty Four)
> Fedora release 44 (Forty Four)
> [root@qdevice ~]#
> [root@qdevice ~]# uname -a
> Linux qdevice 7.1.0-rc5 #1 SMP PREEMPT_DYNAMIC Mon May 25 15:02:20
> EEST 2026 x86_64 GNU/Linux
> [root@qdevice ~]#
> [root@qdevice mdadm]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> nvme6n1 259:1 0 447.1G 0 disk
> ├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
> ├─nvme6n1p2 259:3 0 1G 0 part /boot
> ├─nvme6n1p3 259:4 0 220G 0 part /
> ├─nvme6n1p4 259:5 0 1G 0 part
> └─nvme6n1p5 259:6 0 220G 0 part
> nvme3n1 259:10 0 2.9T 0 disk
> nvme1n1 259:11 0 2.9T 0 disk
> nvme0n1 259:12 0 2.9T 0 disk
> nvme4n1 259:16 0 2.9T 0 disk
> nvme2n1 259:17 0 2.9T 0 disk
> nvme5n1 259:18 0 2.9T 0 disk
> [root@qdevice mdadm]#
> [root@qdevice mdadm]# mdadm --create --verbose /dev/md0 --level=6
> --bitmap=lockless --raid-devices=6 /dev/nvme0n1 /dev/nvme1n1
> /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1
> mdadm: Experimental lockless bitmap, use at your own risk!
> mdadm: layout defaults to left-symmetric
> mdadm: layout defaults to left-symmetric
> mdadm: chunk size defaults to 512K
> mdadm: /dev/nvme0n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme1n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme2n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme3n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme4n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme5n1 appears to be part of a raid array:
> level=raid6 devices=6 ctime=Wed May 20 11:45:38 2026
> mdadm: size set to 3125484544K
> Continue creating array [y/N]? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@qdevice mdadm]#
> [root@qdevice mdadm]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> nvme6n1 259:1 0 447.1G 0 disk
> ├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
> ├─nvme6n1p2 259:3 0 1G 0 part /boot
> ├─nvme6n1p3 259:4 0 220G 0 part /
> ├─nvme6n1p4 259:5 0 1G 0 part
> └─nvme6n1p5 259:6 0 220G 0 part
> nvme3n1 259:10 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme1n1 259:11 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme0n1 259:12 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme4n1 259:16 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme2n1 259:17 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme5n1 259:18 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> [root@qdevice mdadm]#
> [root@qdevice mdadm]# reboot
> [root@qdevice mdadm]#
>
> After reboot,
>
> [root@qdevice ~]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> nvme6n1 259:1 0 447.1G 0 disk
> ├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
> ├─nvme6n1p2 259:3 0 1G 0 part /boot
> ├─nvme6n1p3 259:4 0 220G 0 part /
> ├─nvme6n1p4 259:5 0 1G 0 part
> └─nvme6n1p5 259:6 0 220G 0 part
> nvme2n1 259:9 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme1n1 259:10 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme4n1 259:15 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme0n1 259:16 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme3n1 259:17 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme5n1 259:18 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> [root@qdevice ~]#
>
>
> Anton
>
> пн, 25 мая 2026 г. в 08:32, Yu Kuai <[email protected]>:
> >
> > Hi,
> >
> > 在 2026/5/20 1:28, Anton Gavriliuk 写道:
> > > Hi
> > >
> > > Does md-raid lockless bitmap support server's reboot ?
> >
> > Yes, of course it should support reboot.
> >
> > >
> > > I don't see md-raid6 created with --bitmap=lockless after reboot.
> >
> > Do you check kernel log if there is any failure log? And do you try
> > mdadm -a manually?
> >
> > BTW, pPlease report with your environment and test script in details,
> > especially kernel version and mdadm version.
> >
> > >
> > > Anton
> > >
> > --
> > Thansk,
> > Kuai