Re: Reg thin pool activation failure - transaction id mismatch

On Wed, Sep 18, 2024 at 7:09=E2=80=AFPM Lakshmi Narasimhan Sundararajan
<[email protected]> wrote:
>
> On Wed, Sep 18, 2024 at 4:25=E2=80=AFPM Lakshmi Narasimhan Sundararajan
> <[email protected]> wrote:
> >
> > Hi Team,
> > Good day all!
> >
> > In yet another long testing run, I see a thin pool was unable to get
> > volume group thin pool activated.
> > vgchange -ay <vgname> fails with
> >
> > ```
> > /# vgchange -ay pwx3
> > Thin pool pwx3-pxpool-tpool (253:25) transaction_id is 6, while expecte=
d 24.
> > Thin pool pwx3-pxpool-tpool (253:25) transaction_id is 6, while expecte=
d 24.
> > Thin pool pwx3-pxpool-tpool (253:25) transaction_id is 6, while expecte=
d 24.
> > Thin pool pwx3-pxpool-tpool (253:25) transaction_id is 6, while expecte=
d 24.
> > 3 logical volume(s) in volume group "pwx3" now active
> > ```
> > It had other non thin lvols in the vg that got activated successfully.
> >
> > I tried fixing the pool, by trying the following ``` lvconvert
> > --repair pwx3 ``` but this fails to fix the thin pool.
>
> type: this cmd should be "lvconvert --repair pwx3/pxpool" instead
> pointing to the thin pool in the vg.
>
>
> >
> > There was last configuration in the lvm config backup location.
> > I fixed the transaction id and used `vgcfgrestore --force -f <patched
> > config> pwx3` and this fixed the transaction id issue and vg was now
> > successfully activated.
> >
> >
> > And my follow up questions are:
> > 1/ what exactly is a transaction id mismatch problem?
> > A little more detail about this problem will help me understand this be=
tter.
> >

It's a synchronization method between userland lvm and in-kernel
thin-pool metadata. As Zdenek explained, the id increases by one while
you change the thin-pool layout by creating/deleting a thin volume or
snapshot. The transaction id ensures lvm and kernel are working on the
same page.

> > 2/ Are there any known workflows that are known to create this issue?
> > So I can be careful to avoid such workflows.
> >

Did you create or delete thin volumes while running power cycle tests?
The transaction id might be mismatched by one If there's power loss
just right between the kernel updating the transaction id. In older
versions of lvm,  lvm might not catch the discrepancy in transaction
id and then continue on queuing more undeliverable messages, leading
to greater differences in the values. I believe those kinds of issues
have been addressed in recent lvm versions.

> > 3/ does recovery as above guarantee data in the pool? or is the pool
> > data suspect?

It is totally fine if the volume layout (e.g., number of volumes, the
size, and thin_id) matches that of the on-disk metadata.


> > My environment is as below where the problem got seen:
> > ```
> > # uname -a
> > Linux pwx-ocp-240-201-w6d5l-worker-0-zmtns
> > 5.14.0-427.28.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 19
> > 14:40:47 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
> > ```
> >
> > ```
> > # lvm version LVM version: 2.03.23(2) (2023-11-21) Library version:
> > 1.02.197 (2023-11-21) Driver version: 4.48.0 Configuration:
> > ./configure --build=3Dx86_64-redhat-linux-gnu
> > --host=3Dx86_64-redhat-linux-gnu --program-prefix=3D
> > --disable-dependency-tracking --prefix=3D/usr --exec-prefix=3D/usr
> > --bindir=3D/usr/bin --sbindir=3D/usr/sbin --sysconfdir=3D/etc
> > --datadir=3D/usr/share --includedir=3D/usr/include --libdir=3D/usr/lib6=
4
> > --libexecdir=3D/usr/libexec --localstatedir=3D/var
> > --sharedstatedir=3D/var/lib --mandir=3D/usr/share/man
> > --infodir=3D/usr/share/info --with-default-dm-run-dir=3D/run
> > --with-default-run-dir=3D/run/lvm --with-default-pid-dir=3D/run
> > --with-default-locking-dir=3D/run/lock/lvm --with-usrlibdir=3D/usr/lib6=
4
> > --enable-fsadm --enable-write_install --with-user=3D --with-group=3D
> > --with-device-uid=3D0 --with-device-gid=3D6 --with-device-mode=3D0660
> > --enable-pkgconfig --enable-cmdlib --enable-dmeventd
> > --enable-blkid_wiping --with-udevdir=3D/usr/lib/udev/rules.d
> > --enable-udev_sync --with-thin=3Dinternal --with-cache=3Dinternal
> > --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-dlmcontrol
> > --enable-lvmlockd-sanlock --enable-dbus-service --enable-notify-dbus
> > --enable-dmfilemapd --with-writecache=3Dinternal --with-vdo=3Dinternal
> > --with-vdo-format=3D/usr/bin/vdoformat --with-integrity=3Dinternal
> > --with-default-use-devices-file=3D1 --disable-silent-rules
> > --enable-app-machineid --enable-editline --disable-readline
> > ```
> >
> > ```
> > # cat /etc/os-release
> > NAME=3D"Red Hat Enterprise Linux CoreOS"
> > ID=3D"rhcos"
> > ID_LIKE=3D"rhel fedora"
> > VERSION=3D"416.94.202407231922-0"
> > VERSION_ID=3D"4.16"
> > VARIANT=3D"CoreOS"
> > VARIANT_ID=3Dcoreos
> > PLATFORM_ID=3D"platform:el9"
> > PRETTY_NAME=3D"Red Hat Enterprise Linux CoreOS 416.94.202407231922-0"
> > ANSI_COLOR=3D"0;31"
> > CPE_NAME=3D"cpe:/o:redhat:enterprise_linux:9::baseos::coreos"
> > HOME_URL=3D"https://www.redhat.com/"
> > DOCUMENTATION_URL=3D"https://docs.okd.io/latest/welcome/index.html"
> > BUG_REPORT_URL=3D"https://access.redhat.com/labs/rhir/"
> > REDHAT_BUGZILLA_PRODUCT=3D"OpenShift Container Platform"
> > REDHAT_BUGZILLA_PRODUCT_VERSION=3D"4.16"
> > REDHAT_SUPPORT_PRODUCT=3D"OpenShift Container Platform"
> > REDHAT_SUPPORT_PRODUCT_VERSION=3D"4.16"
> > OPENSHIFT_VERSION=3D"4.16"
> > RHEL_VERSION=3D9.4
> > OSTREE_VERSION=3D"416.94.202407231922-0"
> > ```
> >
> > Best regards
> >
>