Re: [PATCH RFC 0/3] md/raid1: data corruption with serialization

On Wed, Feb 4, 2026 at 11:58 PM Yu Kuai <[email protected]> wrote:
>
> Hi,
>
> 在 2026/2/4 22:58, Xiao Ni 写道:
> > A data corruption can happen when using serialization for raid1.
> > Serialization is not enabled by default. But it looks like there
> > is a data corruption risk if serialization is closed. Because the
> > lower driver can't guarantee the sequence which io is written first.
> > So it's possible that different member disks will have different
> > data for nvme devices. This patch set doesn't open serialization
> > by default.
>
> An important idea is that if there are filesystem on top of raid1/10/5, there
> is no need to consider overlap bio. The only case for serialization is WriteMostly,
> that bio can be returned to user while it's not done for slow disks. And that is
> the reason why serialization is for rdev.

I know the reason why it needs serialization for writemostly case. At
first, I asked the same question if we need to handle overlap if a
filesystem is on raid. I got the answer from AI, filesystem can submit
overlap bios simultaneously, maybe it's the reason why
serialize_policy was introduced? And database is another case which
can write to raid1 directly without a filesystem. So I started this
job. If we don't need to consider overlap, why do we need
serialize_policy? cc guoqing who is the author of serialize_policy
also.

>
> We definitely will not enable serialization by default, because there will
> be huge performance degradation. And I don't think we'll handle the case user

Yes, this is the reason why I don't enable it.

> manage data with raw disk, and write to the same area concurrently. At last, if
> there are still problem with WriteMostly case, please rebase and keep serialization
> for rdev.

The data corruption is:

R1 issued (split into LO and HI)
(LO)                                                              (HI)
R1 write to high-speed disk begins              R1 write to high-speed
disk waits (waits while LO is being written due to an incorrect sector
range)
R1 write to low-speed disk begins
R1 write to high-speed disk completes
High-speed disk wakes up                           R1 write to
high-speed disk begins

R1 write to low-speed disk waits (waits while LO is being written due
to an incorrect sector range)

R1 write to high-speed disk completes
R1 completion notification

R2 issued (split into LO and HI)
R2 write to high-speed disk begins              R2 write to high-speed
disk waits (waits while LO is being written due to an incorrect sector
range)
R2 write to low-speed disk waits (due to R1 being written)
R2 write to high-speed disk completes
High-speed disk wakes up                           R2 write to
high-speed disk begins

R2 write to low-speed disk Wait (R1 is still waiting)

R2 write completed on high-speed disk
R2 completion notification

R1 write completed on slow disk
Slow disk wake-up
R2 write started on slow disk
R2 write completed on slow disk
Slow disk wake-up                                       R2 write
started on slow disk *R2 will be written before R1 here.

R2 write completed on slow disk

Slow disk wake-up

R1 write started on slow disk

Patch03 can fix this problem. But the root cause should be the
sequence getting lock after being woken up. So besides patch03, it is
necessary to ensure that the order of processing I/O matches the order
of adding I/O. If we don't need to consider upper layer can submit
overlap bios simultaneously, I'll fix this based on rdev rathar than
mddev.

Best Regards
Xiao

>
> >
> > Xiao Ni (3):
> >    md: add return value of mddev_create_serial_pool
> >    md/raid1: fix data corruption by moving serialization to mddev level
> >    md/raid1: fix incorrect sector range in serialization
> >
> >   drivers/md/md-bitmap.c |  28 +++++--
> >   drivers/md/md.c        | 171 ++++++++++++++---------------------------
> >   drivers/md/md.h        |  30 ++++----
> >   drivers/md/raid1.c     |  47 ++++++-----
> >   4 files changed, 115 insertions(+), 161 deletions(-)
> >
> --
> Thansk,
> Kuai
>