Re[2]: LVM-thin metadata corruption on Proxmox after RAID-5 issues – thin_repair/thin_d ump fail

Hi Zdenek,

Thanks for the reply!  Here are the missing version number=E2=80=A6

root@proxmox0:~# uname -a
Linux proxmox0 6.8.12-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-5=20
(2024-12-03T10:26Z) x86_64 GNU/Linux

/etc/os-release says "Debian GNU/Linux 12 (bookworm)=E2=80=9D.

root@proxmox0:~# lvm version
   LVM version:     2.03.16(2) (2022-05-18)
   Library version: 1.02.185 (2022-05-18)
   Driver version:  4.48.0
   Configuration:   ./configure --build=3Dx86_64-linux-gnu --prefix=3D/usr=
=20
--includedir=3D$/include --mandir=3D$/share/man=20
--infodir=3D$/share/info --sysconfdir=3D/etc --localstatedir=3D/var=20
--disable-option-checking --disable-silent-rules=20
--libdir=3D$/lib/x86_64-linux-gnu --runstatedir=3D/run=20
--disable-maintainer-mode --disable-dependency-tracking=20
--libdir=3D/lib/x86_64-linux-gnu --sbindir=3D/sbin=20
--with-usrlibdir=3D/usr/lib/x86_64-linux-gnu --with-optimisation=3D-O2=20
--with-cache=3Dinternal --with-device-uid=3D0 --with-device-gid=3D6=20
--with-device-mode=3D0660 --with-default-pid-dir=3D/run=20
--with-default-run-dir=3D/run/lvm --with-default-locking-dir=3D/run/lock/lv=
m=20
--with-thin=3Dinternal --with-thin-check=3D/usr/sbin/thin_check=20
--with-thin-dump=3D/usr/sbin/thin_dump=20
--with-thin-repair=3D/usr/sbin/thin_repair --with-udev-prefix=3D/=20
--enable-applib --enable-blkid_wiping --enable-cmdlib --enable-dmeventd=20
--enable-editline --enable-lvmlockd-dlm --enable-lvmlockd-sanlock=20
--enable-lvmpolld --enable-notify-dbus --enable-pkgconfig=20
--enable-udev_rules --enable-udev_sync --disable-readline

root@proxmox0:~# thin_repair -V
0.9.0

I did try to "apt install --only-upgrade  thin-provisioning-tools=E2=80=9D, =
but=20
it said "thin-provisioning-tools is already the newest version=20
(0.9.0-2)=E2=80=9D.  Maybe I need a better repository?  Here is the=20
sources.list:

deb http://ftp.de.debian.org/debian bookworm main contrib
deb http://ftp.de.debian.org/debian bookworm-updates main contrib
deb http://security.debian.org bookworm-security main contrib
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

The damage was caused by two disks of a six disk raid-5 array going=20
offline for some reason.  They are both back online and the controller=20
rebuilt the raid-5 without any obvious errors.

Thanks,
Ray

------ Original Message ------
From "Zdenek Kabelac" <[email protected]>
To "Ray Davis" <[email protected]>; [email protected]
Date 09.04.2026 22:32:56
Subject Re: LVM-thin metadata corruption on Proxmox after RAID-5 issues=20
=E2=80=93 thin_repair/thin_dump fail

>Dne 09. 04. 26 v 17:55 Ray Davis napsal(a):
>>Hello,
>>
>>I=E2=80=99m looking for advice on an LVM-thin metadata corruption case on =
a Proxmox host. I have stopped further repair attempts and preserved the c=
urrent state for analysis.
>>
>>Environment
>>
>>* Proxmox VE host
>>* Thin pool: `VMDATA0/VMDATA0`
>>* Backing storage had RAID-5 issues involving two disks
>>* After reseating the disks, the RAID came back online, but the thin pool =
would no longer activate
>>>
>>* Created temporary metadata LVs
>>* Used `lvconvert --swapmetadata` to extract the pool metadata
>>* Activated the extracted metadata LV
>>* Preserved a raw image of the extracted metadata
>>* Ran `thin_check`
>>* Ran `thin_repair`
>>* Ran `thin_dump =E2=80=94repair`
>>
>>I can provide access to the full case directory or a tarball over HTTP if =
someone is willing to look at it. I would prefer to share the link private=
ly with anyone interested.
>>
>>My main questions are:
>>
>>1. Is there any remaining offline recovery path worth trying with the sta=
ndard dm-thin/LVM tools?
>
>Hi
>
>Very extensive report - but it looks like the key element here would be -
>what are the versions in use.
>
>
>Kernel ?
>
>lvm2 ?
>
>thin_repair -V  ?
>
>
>
>>2. Does this failure pattern usually indicate that the mapping metadata i=
s beyond recovery by `thin_repair`/`thin_dump`?
>>3. Would it be useful to inspect the raw metadata image further, and if s=
o, what specific tools or commands would you recommend next?
>>
>>I can provide any command output that would be useful.
>
>If you are using latest tools - then authors of this tool will need to hav=
e
>access to the damaged metadata  whether there is something that can be rec=
overed.
>
>It'd also greatly help knowing what kind of 'damange' may have caused this=
.
>
>Regards
>
>Zdenek
>