Re: thin pool powerfail tests and data loss

Hi,

On Wed, Sep 11, 2024 at 12:05=E2=80=AFAM Lakshmi Narasimhan Sundararajan
<[email protected]> wrote:
>
> Hi Ming,
> Thanks for reaching out.
>
> It looks like at the moment, writes, reads and discards are in the
> mix, with block sizes varying from 16K-128K on a thin pool with chunk
> size 64K, pool capacity many TB.
> multiple thin lvols in the pool(200G), and all of them are busy processin=
g IOs.
> This is on a VM, and storage disks are from iscsi SAN, mpath, md array
> acting as lvm pv,  over which a thin pools gets formed.

Does the thin-pool run inside or outside the VM?  and how do the thin
lvols receive IO?
Do those thin lvols receive IOs from nested hypervisors?

> And while in this setup, and IOs in progress, power cycle tests are
> attempted and after recovering end up with data loss.

Do you perform power cycle tests on the host running thin-pool, or the
hosts generating IO workloads?
Could you please explain how you identified the data loss, given there
might be async IO operations?

> Our internal study is still evolving at the moment, I will share more
> as I find more.
> Switching to latest fedora 40 is a challenge for me, whats the next
> distro you may have in mind? Maybe list a few so I can pick whats
> easier my end.
>
> Regards
>
> On Tue, Sep 10, 2024 at 9:23=E2=80=AFPM Ming Hung Tsai <[email protected]>=
 wrote:
> >
> > On Mon, Sep 9, 2024 at 10:53=E2=80=AFPM Lakshmi Narasimhan Sundararajan
> > <[email protected]> wrote:
> > >
> > > On Mon, Sep 9, 2024 at 6:57=E2=80=AFPM Zdenek Kabelac <zdenek.kabelac=
@gmail.com> wrote:
> > > >
> > > > Dne 08. 09. 24 v 3:37 Lakshmi Narasimhan Sundararajan napsal(a):
> > > > > Hi Team,
> > > > > Greetings!
> > > > >
> > > > > In my experiments with thin pool and power cycle tests, I find th=
at
> > > > > the inflight data has not been persisted, even if there is a sync
> > > > > following write that got completed.
> > > > >
> > > > > suppose we have the following bdev stackup:
> > > > > bdevs(multi) -> mpath -> lvm pv -> lvm vg -> lvm thin pool -> mul=
tiple
> > > > > thin vols.
> > > > >
> > > > > Each of the thin vols that has inflight data, has a corresponding=
 sync
> > > > > checkpoint periodically. The expectation is all writes before the=
 sync
> > > > > are persisted to disk and can survive power reset tests.
> > > > >
> > > > > Like below seq on many thin vols on the thin pool.
> > > > > write -> write -> write -> fsync(fd) -> write -> write -> fsync(f=
d) -> ..etc
> > > > >
> > > > > Now my code only does a fsync(fd) on the thin volume (block devic=
e directly).
> > > > > There are no fs mounts on the volume.
> > > > > Is this sufficient or should the thin pool (/dev/mapper/pool) als=
o be
> > > > > synced along?
> > > > > What would be the proper sequence of sync actions to survive a po=
wer reset?
> > > > >
> > > > > I am also trying to validate if this is an issue because mpath is
> > > > > involved in any way(another dm layer in between), but would appre=
ciate
> > > > > some guidance in this
> > > > > regard.
> > > >
> > > > Hi
> > > >
> > > > Could you please retest your issue with  recent lvm2  (>=3D 2.03.24=
) and recent
> > > > linux kernel   >=3D 6.9
> > >
> > > Thank you for your quick attention.
> > >
> > > That is a big jump.
> > > Would you be willing to consider any distro (say OCP latest?), that
> > > will come with a bundle for both lvm+kernel, is there any such
> > > combination you would consider for this test?
> > > Also, is there any way I can check issues/fixes of interest?
> > >
> > > Is my logic using fsync(thin pool device) sufficient to pass power fa=
il tests?
> > > Are there any stronger versions or more logic to ensure power reset t=
est passes?
> > >
> > > Regards
> > >
> > > >
> > > > Regards
> > > >
> > > > Zdenek
> > > >
> > >
> >
> > Hi,
> >
> > Could you please provide more details on the steps you took that led
> > to the issue?
> > Additionally, if possible, it would be helpful to let us know if the
> > issue is reproducible on Fedora 40, with or without the mpath layer.
> >
> >
> > Thanks,
> > Ming-Hung Tsai
> >
> >
>