[REGRESSION] md/raid5: discard submitted during reshape is never completed, submitter hangs in D state forever

Hi,

Since commit 90573092673c ("md/raid5: validate discard support at
request time", v7.2-rc1), a discard bio submitted to a raid5/raid6
array while a reshape is running is dropped without ever being
completed. The submitting task hangs in uninterruptible sleep
indefinitely, and only a reboot clears it. On my machine this turned
the weekly systemd fstrim.timer run into a day-long filesystem-wide
stall that ended in a hard reset.

Mechanism
---------

Before 90573092673c, an array whose members do not support discard
(or without devices_handle_discard_safely set) advertised
max_hw_discard_sectors = 0, so FITRIM and blkdev discards never
reached raid5 at all. Since that commit the array always advertises
UINT_MAX and member support is validated at request time in
raid5_discard_limits(), which correctly completes unsupported bios
with BLK_STS_NOTSUPP and bio_endio().

However, make_discard_request() still contains the reshape guard
that predates request-time validation (620125f2bf8f, "MD: raid5
trim support", v3.7):

	if (mddev->reshape_position != MaxSector)
		/* Skip discard while reshape is happening */
		return;

(drivers/md/raid5.c:5726 in v7.2, still present in current master.)

This returns without calling bio_endio(). The caller treats the bio
as handled:

	if (unlikely(bio_op(bi) == REQ_OP_DISCARD)) {
		make_discard_request(mddev, bi);
		md_write_end(mddev);
		return true;
	}

so nothing ever completes the bio. blkdev_issue_discard() waits in
submit_bio_wait() forever and the submitter is stuck in D state.
Nothing re-queues the bio when the reshape finishes, so reshape
completion does not recover it.

Note that the reshape check runs before raid5_discard_limits(), so
this hits any raid5/6 array under reshape, including arrays whose
members have no discard support at all and which would otherwise be
correctly rejected with BLK_STS_NOTSUPP. That is exactly my case:
before v7.2 the same fstrim run was a silently skipped no-op for
years; the kernel upgrade made it reachable, and the first time it
fired during a reshape it wedged the machine.

Real-world impact observed
--------------------------

Timeline on the affected machine (arm64, v7.2, raid6 of 10 SATA
HDDs, at the time mid-grow from 8 to 10 members, XFS on LVM on top):

- Fri 19:59: mdadm grow/reshape starts.
- Mon 00:40: weekly fstrim.timer fires. fstrim reaches the XFS
  filesystems on the array and hangs in D state on the first
  discard. It stays there for at least the next 21 hours (verified
  via ps; last check was about 21h in).
- While stuck, the FITRIM ioctl holds XFS allocation group / busy
  extent state, so writeback workers and eventually every writer on
  those filesystems pile up behind it. A DVR service wedged
  completely; the machine (also serving as router) later became
  unresponsive and had to be hard-reset.
- md's sync throttling sees the array as idle the whole time (the
  parked bio never generates member I/O), so the stall is invisible
  in /proc/mdstat and cannot be influenced with sync_speed_min/max.

Sample stack of one victim writer (blocked on the inode lock behind
the writeback chain that is itself stuck behind fstrim):

  [<0>] xfs_ilock+0x15c/0x1c0 [xfs]
  [<0>] xfs_vn_update_time+0x1e8/0x268 [xfs]
  [<0>] file_update_time_flags+0x7c/0x140
  [<0>] kiocb_modified+0x3c/0x58
  [<0>] xfs_file_write_checks+0x1ac/0x388 [xfs]
  [<0>] xfs_file_buffered_write+0x13c/0x390 [xfs]
  [<0>] xfs_file_write_iter+0x170/0x308 [xfs]
  [<0>] do_iter_readv_writev+0x110/0x250
  [<0>] vfs_writev+0x12c/0x2c0
  [<0>] do_writev+0x78/0x160
  [<0>] __arm64_sys_writev

I did not capture the fstrim task's own kernel stack before the
machine had to be hard-reset, so the mechanism above is from code
inspection of the discard path rather than a captured stack. I have
a scripted loop-device reproducer (loop-backed raid6, start a grow,
mount XFS, run fstrim; expected result is fstrim permanently in D)
which I have not run yet because the affected machine is a
production box. I am happy to run it in a VM on request, and to
test patches.

The kernel is the v7.2 stable tag plus two local patches for arm64
platform/PCIe bringup that touch nothing in block/ or drivers/md/.

Suggested direction
-------------------

Discards are advisory, so completing the bio instead of dropping it
seems sufficient: bio_endio() with BLK_STS_OK (treat discard as a
no-op during reshape) or BLK_STS_NOTSUPP, mirroring what
raid5_discard_limits() already does three lines below. Waiting for
the reshape to pass, as regular writes do, would also fix the hang
but keeps FITRIM blockable for hours or days on a slow reshape. I
will leave the choice to you.

#regzbot introduced: 90573092673c

Thanks,
Pim Zandbergen