[QUESTION] Using bcache to mask transient I/O hangs and errors from an unstable backing device

Hi all,

I'd like to describe a problem we're facing and a proposed solution based o=
n bcache. I'm writing to ask whether this direction makes sense, and whethe=
r there is a better existing mechanism in the kernel that I might have miss=
ed.

=3D=3D Problem =3D=3D
We use kernel RBD (krbd) as a block device backed by a Ceph cluster. Due to=
 network instability and other infrastructure issues, the krbd device occas=
ionally suffers transient I/O hangs or I/O errors. These episodes can last =
anywhere from a few seconds to several minutes, after which krbd recovers o=
n its own.
During these periods, upper-layer applications (filesystems, databases, etc=
.) observe hung or failed I/O and may degrade or crash. Our goal is to make=
 these transient backing-device failures completely invisible to the applic=
ation layer.
To put it more generally: when a block device is unstable and may experienc=
e intermittent I/O hangs or I/O errors, how can we guarantee I/O stability =
for the layers above it?

=3D=3D Proposed approach =3D=3D
We plan to use bcache with a local NVMe device as the cache, sitting in fro=
nt of the krbd backing device. The idea is to let the NVMe absorb all I/O d=
uring a krbd stall and drain dirty data back to krbd once it recovers. Spec=
ifically:
=C2=A0 - The NVMe cache partition is sized equal to the krbd device, so the=
 entire working set can reside in cache. This maximises read cache hit rate=
. This is the most important.
=C2=A0 - We use writeback mode, so both reads and writes are served from th=
e NVMe first and asynchronously flushed to the krbd backing device.
=C2=A0 - The workload is a mix of reads and writes.
bcache already supports most of what we need. However, the current writebac=
k mode does not fully isolate the upper I/O path from backing-device failur=
es. When krbd hangs or returns errors during dirty-data flushing, bcache ma=
y still propagate those failures upward or stall the cache device.

=3D=3D What we think is needed =3D=3D
We believe a relatively small addition =E2=80=94 a new cache mode alongside=
 the existing write-through / writeback / write-around / none modes =E2=80=
=94 could solve this. The semantics of this new mode would be:
=C2=A0 * All reads and writes are served exclusively from the cache device.
=C2=A0 * Dirty data is flushed to the backing device asynchronously.
=C2=A0 * Any I/O errors or hangs on the backing device during flushing are =
handled gracefully =E2=80=94 retried later rather than propagated to the up=
per layer.
=C2=A0 * When the backing device is healthy, dirty data drains normally.
This would allow bcache to act as a resilience layer, not just a performanc=
e cache. The required changes seem modest and would not affect the existing=
 modes.

=3D=3D Questions =3D=3D
=C2=A0 1) Is this direction sound within the bcache architecture? Is there =
anything fundamental that would make it impractical?
=C2=A0 2) Would adding such a new mode to bcache be considered meaningful a=
nd welcome? I'm willing to do the development and submit patches, but I wan=
t to make sure this is not out of scope for the project.
=C2=A0 3) Is there an existing in-kernel solution =E2=80=94 dm-cache, dm-wr=
itecache, or some other mechanism =E2=80=94 that already handles the "mask =
transient backing-device failures" use case and that I may have overlooked?

Any feedback, pointers, or alternative suggestions would be greatly appreci=
ated. Thank you for your time.

Best regards.