Re: thin pool powerfail tests and data loss

Hi Ming,
Thanks for reaching out.

It looks like at the moment, writes, reads and discards are in the
mix, with block sizes varying from 16K-128K on a thin pool with chunk
size 64K, pool capacity many TB.
multiple thin lvols in the pool(200G), and all of them are busy processing =
IOs.
This is on a VM, and storage disks are from iscsi SAN, mpath, md array
acting as lvm pv,  over which a thin pools gets formed.
And while in this setup, and IOs in progress, power cycle tests are
attempted and after recovering end up with data loss.

Our internal study is still evolving at the moment, I will share more
as I find more.
Switching to latest fedora 40 is a challenge for me, whats the next
distro you may have in mind? Maybe list a few so I can pick whats
easier my end.

Regards

On Tue, Sep 10, 2024 at 9:23=E2=80=AFPM Ming Hung Tsai <[email protected]> w=
rote:
>
> On Mon, Sep 9, 2024 at 10:53=E2=80=AFPM Lakshmi Narasimhan Sundararajan
> <[email protected]> wrote:
> >
> > On Mon, Sep 9, 2024 at 6:57=E2=80=AFPM Zdenek Kabelac <zdenek.kabelac@g=
mail.com> wrote:
> > >
> > > Dne 08. 09. 24 v 3:37 Lakshmi Narasimhan Sundararajan napsal(a):
> > > > Hi Team,
> > > > Greetings!
> > > >
> > > > In my experiments with thin pool and power cycle tests, I find that
> > > > the inflight data has not been persisted, even if there is a sync
> > > > following write that got completed.
> > > >
> > > > suppose we have the following bdev stackup:
> > > > bdevs(multi) -> mpath -> lvm pv -> lvm vg -> lvm thin pool -> multi=
ple
> > > > thin vols.
> > > >
> > > > Each of the thin vols that has inflight data, has a corresponding s=
ync
> > > > checkpoint periodically. The expectation is all writes before the s=
ync
> > > > are persisted to disk and can survive power reset tests.
> > > >
> > > > Like below seq on many thin vols on the thin pool.
> > > > write -> write -> write -> fsync(fd) -> write -> write -> fsync(fd)=
 -> ..etc
> > > >
> > > > Now my code only does a fsync(fd) on the thin volume (block device =
directly).
> > > > There are no fs mounts on the volume.
> > > > Is this sufficient or should the thin pool (/dev/mapper/pool) also =
be
> > > > synced along?
> > > > What would be the proper sequence of sync actions to survive a powe=
r reset?
> > > >
> > > > I am also trying to validate if this is an issue because mpath is
> > > > involved in any way(another dm layer in between), but would appreci=
ate
> > > > some guidance in this
> > > > regard.
> > >
> > > Hi
> > >
> > > Could you please retest your issue with  recent lvm2  (>=3D 2.03.24) =
and recent
> > > linux kernel   >=3D 6.9
> >
> > Thank you for your quick attention.
> >
> > That is a big jump.
> > Would you be willing to consider any distro (say OCP latest?), that
> > will come with a bundle for both lvm+kernel, is there any such
> > combination you would consider for this test?
> > Also, is there any way I can check issues/fixes of interest?
> >
> > Is my logic using fsync(thin pool device) sufficient to pass power fail=
 tests?
> > Are there any stronger versions or more logic to ensure power reset tes=
t passes?
> >
> > Regards
> >
> > >
> > > Regards
> > >
> > > Zdenek
> > >
> >
>
> Hi,
>
> Could you please provide more details on the steps you took that led
> to the issue?
> Additionally, if possible, it would be helpful to let us know if the
> issue is reproducible on Fedora 40, with or without the mpath layer.
>
>
> Thanks,
> Ming-Hung Tsai
>
>