Re: thin pool powerfail tests and data loss

On Wed, Sep 18, 2024 at 6:25=E2=80=AFPM Lakshmi Narasimhan Sundararajan
<[email protected]> wrote:
>
> On Wed, Sep 18, 2024 at 3:51=E2=80=AFPM Ming Hung Tsai <[email protected]>=
 wrote:
> >
> > Hi Lakshimi,
> >
> > I'm glad to hear you've resolved the issue. If you're comfortable
> > sharing, would you mind providing some insights into the root cause?
>
> Why not, fsync(thin dev) got incorrectly built leaving out a few
> devices while syncing a pool with multiple thin devices.
>
> LN
>

Thanks for the update. Was it due to an issue during the kernel build
process or possibly some custom patches? Also, do you still need the
"dmsetup suspend/resume pool" workaround to pass the tests?


>
> >
> >
> > On Wed, Sep 18, 2024 at 1:03=E2=80=AFAM Lakshmi Narasimhan Sundararajan
> > <[email protected]> wrote:
> > >
> > > On Fri, Sep 13, 2024 at 7:49=E2=80=AFPM Tony Asleson <tasleson@redhat=
.com> wrote:
> > > >
> > > > You may want to check out https://lwn.net/Articles/457667/
> > > >
> > > > On Fri, Sep 13, 2024 at 9:06=E2=80=AFAM Zdenek Kabelac <zdenek.kabe=
[email protected]> wrote:
> > > > >
> > > > > Dne 13. 09. 24 v 7:55 Lakshmi Narasimhan Sundararajan napsal(a):
> > > > > > Hi Ming,
> > > > > > I am still collecting results, so I will present findings that =
are
> > > > > > confirmed so far.
> > > > > > There is some good news too.
> > > > > >
> > > > > > On Fri, Sep 13, 2024 at 1:11=E2=80=AFAM Ming Hung Tsai <mtsai@r=
edhat.com> wrote:
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> On Wed, Sep 11, 2024 at 12:05=E2=80=AFAM Lakshmi Narasimhan Su=
ndararajan
> > > > > >> <[email protected]> wrote:
> > > > >
> > > > > > My application that is consuming the thin device pumps IO traff=
ic
> > > > > > directly on the raw block device.
> > > > > > My application also keeps a journal record outside the thin poo=
l and
> > > > > > after power recycled, reading the data back
> > > > > > did not guarantee sync consistency.
> > > > > >
> > > > > > I wrote a sample program that is trying to recreate this outsid=
e my application.
> > > > > > here it is: sulakshm/iotest: iotest (github.com)
> > > > > > I am still refining it, as I have not seen the problem with thi=
s tool yet.
> > > > > > But the logic is similar to how my application consumes thin de=
v; and
> > > > > > my app can reproduce this very easily.
> > > > > >
> > > > > > As I said before, I am still collecting additional information =
from
> > > > > > many internal tests.
> > > > > > So far, I can see this problem even in 6.5 kernel.
> > > > > >
> > > > > > The latest distro/linux kernel where this problem is seen.
> > > > > >> 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Ja=
n 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > > > > >
> > > > > >
> > > > > > There is a workaround to this problem that is looking promising=
, tests
> > > > > > ongoing still.
> > > > > >
> > > > > > The sync point from my application is a sync(fd) of the thin de=
v.
> > > > > > This has proven insufficient.
> > > > > > In addition, I had to perform "dmsetup suspend pool -> dmsetup =
resume pool".
> > > > > > This guarantees sync point consistency.
> > > > > >
> > > > >
> > > > > Hi
> > > > >
> > > > > Not exactly sure what your app is all exactly doing - however the=
re is
> > > > > cut&paste from 'fsync()'  manpage:
> > > > >
> > > > > ---
> > > > > Calling fsync() does not necessarily ensure that the entry in the=
 directory
> > > > > containing the file has also  reached  disk.   For  that  an  exp=
licit
> > > > > fsync() on a file descriptor for the directory is also needed.
> > > > > ---
> > > > >
> > > > > For this purpose our 'test suite' app basically  'opens' whole de=
vice and
> > > > > fsync and close it - to ensure synchronization point flush.
> > > > >
> > > > > Thus 'suspend & resume' of the whole thin device could be possibl=
y unnecessary
> > > > > - just do a fsync() on blockdevice fd.
> > > > >
> > > > >
> > > > > You can also play fun games with 'fsfreeze' operation.
> > >
> > >
> > > Good day all!
> > >
> > > This issue has been rootcaused successfully to an issue with my appli=
cation.
> > > It had been a tough last week given the nature of the issue, and
> > > thanks for everyone who reached out with helpful suggestions.
> > > As part of this process and bug verification, I would gladly submit
> > > that the thin pool implementation did withstand the power cycle tests
> > > wonderfully.
> > >
> > > Best regards and you all have a wonderful day.
> > >
> > >
> > > > >
> > > > > Regards
> > > > >
> > > > > Zdenek
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>