Re: [PATCH RFC 0/3] md/raid1: data corruption with serialization

On Thu, Feb 5, 2026 at 8:34 AM Xiao Ni <[email protected]> wrote:
>
> On Wed, Feb 4, 2026 at 11:58 PM Yu Kuai <[email protected]> wrote:
> >
> > Hi,
> >
> > 在 2026/2/4 22:58, Xiao Ni 写道:
> > > A data corruption can happen when using serialization for raid1.
> > > Serialization is not enabled by default. But it looks like there
> > > is a data corruption risk if serialization is closed. Because the
> > > lower driver can't guarantee the sequence which io is written first.
> > > So it's possible that different member disks will have different
> > > data for nvme devices. This patch set doesn't open serialization
> > > by default.
> >
> > An important idea is that if there are filesystem on top of raid1/10/5, there
> > is no need to consider overlap bio. The only case for serialization is WriteMostly,
> > that bio can be returned to user while it's not done for slow disks. And that is
> > the reason why serialization is for rdev.
>
> I know the reason why it needs serialization for writemostly case. At
> first, I asked the same question if we need to handle overlap if a
> filesystem is on raid. I got the answer from AI, filesystem can submit
> overlap bios simultaneously, maybe it's the reason why
> serialize_policy was introduced? And database is another case which
> can write to raid1 directly without a filesystem. So I started this
> job. If we don't need to consider overlap, why do we need
> serialize_policy? cc guoqing who is the author of serialize_policy
> also.
>
> >
> > We definitely will not enable serialization by default, because there will
> > be huge performance degradation. And I don't think we'll handle the case user
>
> Yes, this is the reason why I don't enable it.
>
> > manage data with raw disk, and write to the same area concurrently. At last, if
> > there are still problem with WriteMostly case, please rebase and keep serialization
> > for rdev.
>
> The data corruption is:
>
> R1 issued (split into LO and HI)
> (LO)                                                              (HI)
> R1 write to high-speed disk begins              R1 write to high-speed
> disk waits (waits while LO is being written due to an incorrect sector
> range)
> R1 write to low-speed disk begins
> R1 write to high-speed disk completes
> High-speed disk wakes up                           R1 write to
> high-speed disk begins
>
> R1 write to low-speed disk waits (waits while LO is being written due
> to an incorrect sector range)
>
> R1 write to high-speed disk completes
> R1 completion notification
>
> R2 issued (split into LO and HI)
> R2 write to high-speed disk begins              R2 write to high-speed
> disk waits (waits while LO is being written due to an incorrect sector
> range)
> R2 write to low-speed disk waits (due to R1 being written)
> R2 write to high-speed disk completes
> High-speed disk wakes up                           R2 write to
> high-speed disk begins
>
> R2 write to low-speed disk Wait (R1 is still waiting)
>
> R2 write completed on high-speed disk
> R2 completion notification
>
> R1 write completed on slow disk
> Slow disk wake-up
> R2 write started on slow disk
> R2 write completed on slow disk
> Slow disk wake-up                                       R2 write
> started on slow disk *R2 will be written before R1 here.
>
> R2 write completed on slow disk
>
> Slow disk wake-up
>
> R1 write started on slow disk

Sorry the format looks a mess.

And let me change to this:
Setup: RAID1 with one fast disk and one slow disk. Two requests R1 and
R2, each split into LO and HI parts

Step 1-10: Request R1 Processing
  - R1-LO: writes to fast disk → completes
  - R1-LO: writes to slow disk → starts (slow)
  - R1-HI: incorrectly waits for R1-LO (wrong sector range check)
  - R1-HI: R1-LO finishes and wakes up serial_io_wait. R1-HI finally
writes to fast disk → completes
  - R1 completion notified (but R1-LO to slow disk is still handling,
R1-HI still pending on slow disk)

Step 11-19: Request R2 Processing
  - R2-LO: writes to fast disk → completes
  - R2-LO: waits for R1-LO on slow disk (correct behavior)
  - R2-HI: incorrectly waits for R2-LO (wrong sector range check)
  - R2-HI: writes to fast disk → completes
  - R2 completion notified

Now, R1-LO is writing to slow disk, R2-LO is pending on slow disk.
R1-HI and R2-HI are pending on slow disk

Step 20-28: Slow Disk Completes (OUT OF ORDER!)
  - R1-LO completes on slow disk
  - R2-LO starts and completes on slow disk
  - R2-HI starts and completes on slow disk ← R2 BEFORE R1!
  - R1-HI finally starts and completes on slow disk

RESULT:
=======
Fast disk write order:  R1-LO → R1-HI → R2-LO → R2-HI  (CORRECT)
Slow disk write order:  R1-LO → R2-LO → R2-HI → R1-HI  (WRONG!)

The slow disk has R2-HI written before R1-HI, causing data corruption.

Best Regards
Xiao


>
> Patch03 can fix this problem. But the root cause should be the
> sequence getting lock after being woken up. So besides patch03, it is
> necessary to ensure that the order of processing I/O matches the order
> of adding I/O. If we don't need to consider upper layer can submit
> overlap bios simultaneously, I'll fix this based on rdev rathar than
> mddev.
>
> Best Regards
> Xiao
>
> >
> > >
> > > Xiao Ni (3):
> > >    md: add return value of mddev_create_serial_pool
> > >    md/raid1: fix data corruption by moving serialization to mddev level
> > >    md/raid1: fix incorrect sector range in serialization
> > >
> > >   drivers/md/md-bitmap.c |  28 +++++--
> > >   drivers/md/md.c        | 171 ++++++++++++++---------------------------
> > >   drivers/md/md.h        |  30 ++++----
> > >   drivers/md/raid1.c     |  47 ++++++-----
> > >   4 files changed, 115 insertions(+), 161 deletions(-)
> > >
> > --
> > Thansk,
> > Kuai
> >