Re: thin pool powerfail tests and data loss

On Wed, Sep 18, 2024 at 4:14=E2=80=AFPM Ming Hung Tsai <[email protected]> w=
rote:
>
> On Wed, Sep 18, 2024 at 6:25=E2=80=AFPM Lakshmi Narasimhan Sundararajan
> <[email protected]> wrote:
> >
> > On Wed, Sep 18, 2024 at 3:51=E2=80=AFPM Ming Hung Tsai <[email protected]=
m> wrote:
> > >
> > > Hi Lakshimi,
> > >
> > > I'm glad to hear you've resolved the issue. If you're comfortable
> > > sharing, would you mind providing some insights into the root cause?
> >
> > Why not, fsync(thin dev) got incorrectly built leaving out a few
> > devices while syncing a pool with multiple thin devices.
> >
> > LN
> >
>
> Thanks for the update. Was it due to an issue during the kernel build
> process or possibly some custom patches? Also, do you still need the
> "dmsetup suspend/resume pool" workaround to pass the tests?



My _userspace_ application internally does build this sync list, and
due to a bug incorrectly, left some devices off.
No kernel layer issue was uncovered, and trying to recreate this
outside my application, as I already pointed out in the past,
was unsuccessful.

LN
>
>
> >
> > >
> > >
> > > On Wed, Sep 18, 2024 at 1:03=E2=80=AFAM Lakshmi Narasimhan Sundararaj=
an
> > > <[email protected]> wrote:
> > > >
> > > > On Fri, Sep 13, 2024 at 7:49=E2=80=AFPM Tony Asleson <tasleson@redh=
at.com> wrote:
> > > > >
> > > > > You may want to check out https://lwn.net/Articles/457667/
> > > > >
> > > > > On Fri, Sep 13, 2024 at 9:06=E2=80=AFAM Zdenek Kabelac <zdenek.ka=
[email protected]> wrote:
> > > > > >
> > > > > > Dne 13. 09. 24 v 7:55 Lakshmi Narasimhan Sundararajan napsal(a)=
:
> > > > > > > Hi Ming,
> > > > > > > I am still collecting results, so I will present findings tha=
t are
> > > > > > > confirmed so far.
> > > > > > > There is some good news too.
> > > > > > >
> > > > > > > On Fri, Sep 13, 2024 at 1:11=E2=80=AFAM Ming Hung Tsai <mtsai=
@redhat.com> wrote:
> > > > > > >>
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> On Wed, Sep 11, 2024 at 12:05=E2=80=AFAM Lakshmi Narasimhan =
Sundararajan
> > > > > > >> <[email protected]> wrote:
> > > > > >
> > > > > > > My application that is consuming the thin device pumps IO tra=
ffic
> > > > > > > directly on the raw block device.
> > > > > > > My application also keeps a journal record outside the thin p=
ool and
> > > > > > > after power recycled, reading the data back
> > > > > > > did not guarantee sync consistency.
> > > > > > >
> > > > > > > I wrote a sample program that is trying to recreate this outs=
ide my application.
> > > > > > > here it is: sulakshm/iotest: iotest (github.com)
> > > > > > > I am still refining it, as I have not seen the problem with t=
his tool yet.
> > > > > > > But the logic is similar to how my application consumes thin =
dev; and
> > > > > > > my app can reproduce this very easily.
> > > > > > >
> > > > > > > As I said before, I am still collecting additional informatio=
n from
> > > > > > > many internal tests.
> > > > > > > So far, I can see this problem even in 6.5 kernel.
> > > > > > >
> > > > > > > The latest distro/linux kernel where this problem is seen.
> > > > > > >> 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri =
Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > > > > > >
> > > > > > >
> > > > > > > There is a workaround to this problem that is looking promisi=
ng, tests
> > > > > > > ongoing still.
> > > > > > >
> > > > > > > The sync point from my application is a sync(fd) of the thin =
dev.
> > > > > > > This has proven insufficient.
> > > > > > > In addition, I had to perform "dmsetup suspend pool -> dmsetu=
p resume pool".
> > > > > > > This guarantees sync point consistency.
> > > > > > >
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > Not exactly sure what your app is all exactly doing - however t=
here is
> > > > > > cut&paste from 'fsync()'  manpage:
> > > > > >
> > > > > > ---
> > > > > > Calling fsync() does not necessarily ensure that the entry in t=
he directory
> > > > > > containing the file has also  reached  disk.   For  that  an  e=
xplicit
> > > > > > fsync() on a file descriptor for the directory is also needed.
> > > > > > ---
> > > > > >
> > > > > > For this purpose our 'test suite' app basically  'opens' whole =
device and
> > > > > > fsync and close it - to ensure synchronization point flush.
> > > > > >
> > > > > > Thus 'suspend & resume' of the whole thin device could be possi=
bly unnecessary
> > > > > > - just do a fsync() on blockdevice fd.
> > > > > >
> > > > > >
> > > > > > You can also play fun games with 'fsfreeze' operation.
> > > >
> > > >
> > > > Good day all!
> > > >
> > > > This issue has been rootcaused successfully to an issue with my app=
lication.
> > > > It had been a tough last week given the nature of the issue, and
> > > > thanks for everyone who reached out with helpful suggestions.
> > > > As part of this process and bug verification, I would gladly submit
> > > > that the thin pool implementation did withstand the power cycle tes=
ts
> > > > wonderfully.
> > > >
> > > > Best regards and you all have a wonderful day.
> > > >
> > > >
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > Zdenek
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>