Re: thin pool powerfail tests and data loss

Hi Ming,
I am still collecting results, so I will present findings that are
confirmed so far.
There is some good news too.

On Fri, Sep 13, 2024 at 1:11=E2=80=AFAM Ming Hung Tsai <[email protected]> w=
rote:
>
> Hi,
>
> On Wed, Sep 11, 2024 at 12:05=E2=80=AFAM Lakshmi Narasimhan Sundararajan
> <[email protected]> wrote:
> >
> > Hi Ming,
> > Thanks for reaching out.
> >
> > It looks like at the moment, writes, reads and discards are in the
> > mix, with block sizes varying from 16K-128K on a thin pool with chunk
> > size 64K, pool capacity many TB.
> > multiple thin lvols in the pool(200G), and all of them are busy process=
ing IOs.
> > This is on a VM, and storage disks are from iscsi SAN, mpath, md array
> > acting as lvm pv,  over which a thin pools gets formed.
>
> Does the thin-pool run inside or outside the VM?  and how do the thin
> lvols receive IO?
> Do those thin lvols receive IOs from nested hypervisors?

node instance may not be VM; sorry if that additional detail misled you.
My application consumes block devices (thin devices) from the thin pool.
Data on the block device is immaterial.
My application performs IO operations directly to the thin device.


>
> > And while in this setup, and IOs in progress, power cycle tests are
> > attempted and after recovering end up with data loss.
>
> Do you perform power cycle tests on the host running thin-pool, or the
> hosts generating IO workloads?
> Could you please explain how you identified the data loss, given there
> might be async IO operations?

My application that is consuming the thin device pumps IO traffic
directly on the raw block device.
My application also keeps a journal record outside the thin pool and
after power recycled, reading the data back
did not guarantee sync consistency.

I wrote a sample program that is trying to recreate this outside my applica=
tion.
here it is: sulakshm/iotest: iotest (github.com)
I am still refining it, as I have not seen the problem with this tool yet.
But the logic is similar to how my application consumes thin dev; and
my app can reproduce this very easily.

As I said before, I am still collecting additional information from
many internal tests.
So far, I can see this problem even in 6.5 kernel.

The latest distro/linux kernel where this problem is seen.
> 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:=
30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux


There is a workaround to this problem that is looking promising, tests
ongoing still.

The sync point from my application is a sync(fd) of the thin dev.
This has proven insufficient.
In addition, I had to perform "dmsetup suspend pool -> dmsetup resume pool"=
.
This guarantees sync point consistency.

I believe at this point, there are still some operations IOs that are
in the pool that are dependent on the thin dev and are not flushed to
disk.
Doing the above workaround does the additional job and guarantees the
sync consistency.

I wanted to write this summary with more data points, since I see you
are tracking and want to make progress,
I am giving out this early information.

I will follow up with additional notes, but from your end, I would
appreciate it if you could help with the issue and see how this
workaround and my initial analysis as to the likely reason could cause
this.

Regards

>
> > Our internal study is still evolving at the moment, I will share more
> > as I find more.
> > Switching to latest fedora 40 is a challenge for me, whats the next
> > distro you may have in mind? Maybe list a few so I can pick whats
> > easier my end.
> >
> > Regards
> >
> > On Tue, Sep 10, 2024 at 9:23=E2=80=AFPM Ming Hung Tsai <[email protected]=
m> wrote:
> > >
> > > On Mon, Sep 9, 2024 at 10:53=E2=80=AFPM Lakshmi Narasimhan Sundararaj=
an
> > > <[email protected]> wrote:
> > > >
> > > > On Mon, Sep 9, 2024 at 6:57=E2=80=AFPM Zdenek Kabelac <zdenek.kabel=
[email protected]> wrote:
> > > > >
> > > > > Dne 08. 09. 24 v 3:37 Lakshmi Narasimhan Sundararajan napsal(a):
> > > > > > Hi Team,
> > > > > > Greetings!
> > > > > >
> > > > > > In my experiments with thin pool and power cycle tests, I find =
that
> > > > > > the inflight data has not been persisted, even if there is a sy=
nc
> > > > > > following write that got completed.
> > > > > >
> > > > > > suppose we have the following bdev stackup:
> > > > > > bdevs(multi) -> mpath -> lvm pv -> lvm vg -> lvm thin pool -> m=
ultiple
> > > > > > thin vols.
> > > > > >
> > > > > > Each of the thin vols that has inflight data, has a correspondi=
ng sync
> > > > > > checkpoint periodically. The expectation is all writes before t=
he sync
> > > > > > are persisted to disk and can survive power reset tests.
> > > > > >
> > > > > > Like below seq on many thin vols on the thin pool.
> > > > > > write -> write -> write -> fsync(fd) -> write -> write -> fsync=
(fd) -> ..etc
> > > > > >
> > > > > > Now my code only does a fsync(fd) on the thin volume (block dev=
ice directly).
> > > > > > There are no fs mounts on the volume.
> > > > > > Is this sufficient or should the thin pool (/dev/mapper/pool) a=
lso be
> > > > > > synced along?
> > > > > > What would be the proper sequence of sync actions to survive a =
power reset?
> > > > > >
> > > > > > I am also trying to validate if this is an issue because mpath =
is
> > > > > > involved in any way(another dm layer in between), but would app=
reciate
> > > > > > some guidance in this
> > > > > > regard.
> > > > >
> > > > > Hi
> > > > >
> > > > > Could you please retest your issue with  recent lvm2  (>=3D 2.03.=
24) and recent
> > > > > linux kernel   >=3D 6.9
> > > >
> > > > Thank you for your quick attention.
> > > >
> > > > That is a big jump.
> > > > Would you be willing to consider any distro (say OCP latest?), that
> > > > will come with a bundle for both lvm+kernel, is there any such
> > > > combination you would consider for this test?
> > > > Also, is there any way I can check issues/fixes of interest?
> > > >
> > > > Is my logic using fsync(thin pool device) sufficient to pass power =
fail tests?
> > > > Are there any stronger versions or more logic to ensure power reset=
 test passes?
> > > >
> > > > Regards
> > > >
> > > > >
> > > > > Regards
> > > > >
> > > > > Zdenek
> > > > >
> > > >
> > >
> > > Hi,
> > >
> > > Could you please provide more details on the steps you took that led
> > > to the issue?
> > > Additionally, if possible, it would be helpful to let us know if the
> > > issue is reproducible on Fedora 40, with or without the mpath layer.
> > >
> > >
> > > Thanks,
> > > Ming-Hung Tsai
> > >
> > >
> >
>