Re[2]: LVM-thin metadata corruption on Proxmox after RAID-5 issues – thin_repair/thin_d ump fail

Hi Ming-Hung,

Thanks for the reply!  The thin-provisioning-tools are version 0.9.0-2.
Maybe I need a better repository? Here is the sources.list:

deb http://ftp.de.debian.org/debian bookworm main contrib
deb http://ftp.de.debian.org/debian bookworm-updates main contrib
deb http://security.debian.org bookworm-security main contrib
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

The raw metadata is about 17 GB.  Would it be better for me to make it=20
available for download or just let you log into the machine directly?

Thanks,
Ray


------ Original Message ------
From "Ming-Hung Tsai" <[email protected]>
To "Ray Davis" <[email protected]>
Cc [email protected]
Date 10.04.2026 05:43:35
Subject Re: LVM-thin metadata corruption on Proxmox after RAID-5 issues=20
=E2=80=93 thin_repair/thin_dump fail

>On Fri, Apr 10, 2026 at 12:08=E2=80=AFAM Ray Davis <[email protected]> wrote:
>>
>>  Hello,
>>
>>  I=E2=80=99m looking for advice on an LVM-thin metadata corruption case=
 on a
>>  Proxmox host. I have stopped further repair attempts and preserved the
>>  current state for analysis.
>>
>>  Environment
>>
>>  * Proxmox VE host
>>  * Thin pool: `VMDATA0/VMDATA0`
>>  * Backing storage had RAID-5 issues involving two disks
>>  * After reseating the disks, the RAID came back online, but the thin
>>  pool would no longer activate
>>
>>  Original Proxmox/LVM error
>>  `activating LV 'VMDATA0/VMDATA0' failed: Check of pool VMDATA0/VMDATA0
>>  failed (status:1). Manual repair required!`
>>
>>  What I tried
>>
>>  1. `vgcfgbackup VMDATA0`
>>  2. `lvconvert --repair VMDATA0/VMDATA0`
>>
>>  This failed with:
>>  `value size mismatch: expected 8, but got 24 (block 13182)`
>>
>>  At that point I added temporary VG space on a USB disk so I could try
>>  manual metadata recovery.
>>
>>  Manual recovery steps attempted
>>
>>  * Created temporary metadata LVs
>>  * Used `lvconvert --swapmetadata` to extract the pool metadata
>>  * Activated the extracted metadata LV
>>  * Preserved a raw image of the extracted metadata
>>  * Ran `thin_check`
>>  * Ran `thin_repair`
>>  * Ran `thin_dump =E2=80=94repair`
>>
>>  Current results
>>
>>  `thin_check /dev/VMDATA0/meta_extract` reports:
>>
>>  * `missing devices: [0, -]`
>>  * `bad checksum in btree node (block 79511)`
>>  * `missing all mappings for devices: [0, -]`
>>  * `bad checksum in btree node (block 79506)`
>>
>>  `thin_repair -i /dev/VMDATA0/meta_extract -o /dev/VMDATA0/repair_meta`
>>  fails with:
>>  `value size mismatch: expected 8, but got 24 (block 13182)`
>>
>>  `thin_dump --repair -o /root/VMDATA0_repaired.xml
>>  /dev/VMDATA0/meta_extract` fails with the same:
>>  `value size mismatch: expected 8, but got 24 (block 13182)`
>>
>>  Current LV state at the time of extraction looked like this:
>>
>>  * thin pool `VMDATA0`
>>  * extracted metadata LV `meta_extract`
>>  * fresh target metadata LV `repair_meta`
>>
>>  What I have preserved
>>
>>  * Raw metadata image from the extracted metadata LV:
>>     `VMDATA0_meta_extract.raw`
>>  * A second raw copy from the extracted LV device
>>  * `vgcfgbackup` output and LVM archive files
>>  * Diagnostics bundle with:
>>
>>     * `pvs`, `vgs`, `lvs`
>>     * `thin_check` output
>>     * `dmesg`
>>     * kernel journal
>>     * `lvm version`
>>     * checksums
>>
>>  I can provide access to the full case directory or a tarball over HTTP
>>  if someone is willing to look at it. I would prefer to share the link
>>  privately with anyone interested.
>>
>>  My main questions are:
>>
>>  1. Is there any remaining offline recovery path worth trying with the
>>  standard dm-thin/LVM tools?
>>  2. Does this failure pattern usually indicate that the mapping metadata
>>  is beyond recovery by `thin_repair`/`thin_dump`?
>>  3. Would it be useful to inspect the raw metadata image further, and if
>>  so, what specific tools or commands would you recommend next?
>>
>>  I can provide any command output that would be useful.
>>
>>  Thanks very much for any guidance,
>>  Ray
>>
>
>Hi,
>
>The specific error you mentioned, "value size mismatch: expected 8,
>but got 24 (block 13182)", suggests it's an older version of
>thin-provisioning-tools. Hopefully, a newer version will help resolve
>this.
>
>Would you mind providing the raw metadata image for me to look into? Thank =
you.
>
>
>Hank