Re: thin pool powerfail tests and data loss

On Fri, Sep 13, 2024 at 7:49=E2=80=AFPM Tony Asleson <[email protected]> =
wrote:
>
> You may want to check out https://lwn.net/Articles/457667/
>
> On Fri, Sep 13, 2024 at 9:06=E2=80=AFAM Zdenek Kabelac <zdenek.kabelac@gm=
ail.com> wrote:
> >
> > Dne 13. 09. 24 v 7:55 Lakshmi Narasimhan Sundararajan napsal(a):
> > > Hi Ming,
> > > I am still collecting results, so I will present findings that are
> > > confirmed so far.
> > > There is some good news too.
> > >
> > > On Fri, Sep 13, 2024 at 1:11=E2=80=AFAM Ming Hung Tsai <mtsai@redhat.=
com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> On Wed, Sep 11, 2024 at 12:05=E2=80=AFAM Lakshmi Narasimhan Sundarar=
ajan
> > >> <[email protected]> wrote:
> >
> > > My application that is consuming the thin device pumps IO traffic
> > > directly on the raw block device.
> > > My application also keeps a journal record outside the thin pool and
> > > after power recycled, reading the data back
> > > did not guarantee sync consistency.
> > >
> > > I wrote a sample program that is trying to recreate this outside my a=
pplication.
> > > here it is: sulakshm/iotest: iotest (github.com)
> > > I am still refining it, as I have not seen the problem with this tool=
 yet.
> > > But the logic is similar to how my application consumes thin dev; and
> > > my app can reproduce this very easily.
> > >
> > > As I said before, I am still collecting additional information from
> > > many internal tests.
> > > So far, I can see this problem even in 6.5 kernel.
> > >
> > > The latest distro/linux kernel where this problem is seen.
> > >> 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 1=
8:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > >
> > > There is a workaround to this problem that is looking promising, test=
s
> > > ongoing still.
> > >
> > > The sync point from my application is a sync(fd) of the thin dev.
> > > This has proven insufficient.
> > > In addition, I had to perform "dmsetup suspend pool -> dmsetup resume=
 pool".
> > > This guarantees sync point consistency.
> > >
> >
> > Hi
> >
> > Not exactly sure what your app is all exactly doing - however there is
> > cut&paste from 'fsync()'  manpage:
> >
> > ---
> > Calling fsync() does not necessarily ensure that the entry in the direc=
tory
> > containing the file has also  reached  disk.   For  that  an  explicit
> > fsync() on a file descriptor for the directory is also needed.
> > ---
> >
> > For this purpose our 'test suite' app basically  'opens' whole device a=
nd
> > fsync and close it - to ensure synchronization point flush.
> >
> > Thus 'suspend & resume' of the whole thin device could be possibly unne=
cessary
> > - just do a fsync() on blockdevice fd.
> >
> >
> > You can also play fun games with 'fsfreeze' operation.


Good day all!

This issue has been rootcaused successfully to an issue with my application=
.
It had been a tough last week given the nature of the issue, and
thanks for everyone who reached out with helpful suggestions.
As part of this process and bug verification, I would gladly submit
that the thin pool implementation did withstand the power cycle tests
wonderfully.

Best regards and you all have a wonderful day.




> >
> > Regards
> >
> > Zdenek
> >
> >
> >
>