Re: [QUESTION] Debugging some file data corruption

On 2026/01/13 3:11, Calvin Owens wrote:
> On Wednesday 11/12 at 07:32 +1030, Qu Wenruo wrote:
>> 在 2025/11/12 03:31, Calvin Owens 写道:
>>> Hello all,
>>>
>>> I'm looking for help debugging some corruption I recently encountered.
>>> It happened on 6.17.0, and I'm trying to reproduce it on 6.18-rc. This
>>> is not really actionable yet, I'm just looking for advice.
>>>
>>> After copying about 10TB of data to a btrfs+luks+mdraid1 across two 18TB
>>> drives,
>>
>> With LUKS in the middle, it makes any corruption pattern very human
>> unreadable.
>>
>> I guess it's not really feasible to try to reproduce the problem again since
>> it has 10TiB data involved?
>>
>> But if you can spend a lot of time waiting for data copy, mind to try
>> the following combination(s)?
>>
>> - btrfs on mdraid1
>> - btrfs RAID1 on raw two HDDs
> 
> I re-ran the copies several times and never reproduced it. But I finally
> hit it again after I gave up, while making a backup on 6.18.0 with
> btrfs-raid1+luks:
> 
>     Opening filesystem to check...
>     Checking filesystem on /dev/mapper/sdc_crypt
>     UUID: f8223856-32cc-4dcf-8cee-9312e032c005
>     [1/8] checking log skipped (none written)
>     [1/7] checking root items                      (0:00:59 elapsed, 4875792 items checked)
>     [2/7] checking extents                         (0:28:03 elapsed, 4583919 items checked)
>     [3/7] checking free space tree                 (0:01:01 elapsed, 8253 items checked)
>     [4/7] checking fs roots                        (0:20:30 elapsed, 10978 items checked)
>     mirror 2 bytenr 870744010752 csum 0xdb1b27f0ca1a0139f7c65a0c0698a9a3f9e6ca6d624da7f70eecb3fc0f14ffc7 expected csum 0x2bce1cca32d98c3f83087f09980770a101b0560b1ddde7919fbbcd58a75f7d6b
>     mirror 1 bytenr 2004063948800 csum 0xe3f0a16cc8f03705a89f81178d4617c2847d660b7171abe29b65b5b394a9aace expected csum 0xc85b7f37fb620e1a68754692fd7ca43846ca316de6485fc8a4b447bfeab78d78
>     mirror 2 bytenr 3575828525056 csum 0xb9d9ee193d29b59b2015efed6151029c340a051c0338ec7ebca200363d304be8 expected csum 0xd645b7e5add1aa540a3440b78b7d31daa9545c925d32d2650d6bc61b7fdf4813
>     mirror 1 bytenr 3714124914688 csum 0xcb2ff575e84a8e965b94bc8faf9e76d8f645742ee9cd503609efd78ac22623e4 expected csum 0x966b1d021450ffb6c47759161533e185f06a14a50c30ff881097a43b7ad6d6cc
>     mirror 2 bytenr 4211891310592 csum 0xd1c11dabfa4bf3acea463479ab444bbc4c66dc9ba3257f09f4e1815bb46afac2 expected csum 0x17fff4d14269c69d84d267466c577ced3787fd9a9a445e36642067c47f129601
>     mirror 2 bytenr 4328552914944 csum 0x02f75c04f1d921ce34f5b6b9bf40c3b0056971fc89989dad227fc45723938472 expected csum 0xdd6416d001ac47f16e68a24d1438244152355a0774142d4885e5a031c6938d93
>     mirror 2 bytenr 4681011163136 csum 0x44c0b9d90fb258659e7377e93a96039afcadb501559a0cd831bf8c36f8fa1b2b expected csum 0x36c59223538807fc604537be5d686900031866626f3a1dc788755e92a74869cb
>     mirror 1 bytenr 4808263344128 csum 0xfd6278ce98b1f15c8aefea981e5ba7a521fa1a08d0f642185abf72e215288618 expected csum 0x1108b8b604c5b21f7d7d80d32a1fd9c0c0e753cad8fb97967dfa3525105bf808
>     mirror 1 bytenr 4993017057280 csum 0x033440e421102ec0c7b3057eae89dd80f300aecba70d3f1fcf5fe81c2cd6faba expected csum 0xb0e94e343b7291df9aae42a079bd0bba307a1c2ab81315361a49b0d8b6d53f49
>     mirror 2 bytenr 5037246173184 csum 0x00b807ff8d0a31a79bb947e774c077916b0bd162dbf24fe43e4fca179e364214 expected csum 0x984635996e4c3fed6c701519e9f654b1b0adb06f3921522f6b687bbccd730271
>     mirror 1 bytenr 5316786614272 csum 0x2805b36b130667ffaf187f1cc6bdab0802e5eb26332f3bae572202c9c34585dd expected csum 0xd7a2bab5bfddacddb610c6044504474c88962ed22d837a881faba8e4e60bee40
>     mirror 2 bytenr 7293991006208 csum 0x68f917b93233bb6c7e49775c0c0deed66c47fe0027f3d4287a3b2c736fb25a1b expected csum 0x3662cf0b4a35cd0a0b71a475f35a92a6fefdcf21b7ee59a97b261bb6fbda1c8a
>     [5/7] checking csums against data              (33:30:05 elapsed, 6082759 items checked)
>     ERROR: errors found in csum tree
>     [6/7] checking root refs                       (0:00:00 elapsed, 3 items checked)
>     [8/8] checking quota groups skipped (not enabled on this FS)
>     found 8848080683008 bytes used, error(s) found
>     total csum bytes: 68538889856
>     total tree bytes: 75102781440
>     total fs tree bytes: 180338688
>     total extent tree bytes: 340852736
>     btree space waste bytes: 5344101189
>     file data blocks allocated: 8776199127040
>      referenced 8772977901568
> 
> The corruptions looked similar to the first time, couldn't get any new
> clues out of them. Unfortunately it was on LUKS again because I'd given
> up on reproducing this, I'll try without LUKS going forward.
> 
> I realized the common factor betwen the original repro and this one was
> that I was additionally running an sftp copy of the same files over the
> network while the files were being copied between the two local volumes.
> 
>     mkfs.btrfs -m raid1 -d raid1 --csum blake2 /dev/sda /dev/sdb
>     mount /dev/sda -o compress=zstd:1 /mnt
>     rsync -Pav /data/ /mnt
> 
> ...and then, concurrently from another machine:
> 
>     sftp -ar nas:/data .
> 
> Can anybody else reproduce this corruption with that combo?
> 
> Otherwise, I'll keep working on narrowing this down: the tests take ages
> to run but require very little actual human time from me, so I'm happy
> to keep trying. Any other suggestions are welcome!

Hi Calvin,

How about replacing one of your hard drives with a product from a completely 
different manufacturer for isolate the issue?
I've seen this type of problem occur twice due to hardware, 
and neither of them were using btrfs, md or luks.

The first case was about 12 years ago. I had a RAID-Z2 (RAID6) configured 
with ZFS on Linux. There was a problem only with the HDDs connected to the 
Marvell 88SE9128, and the problem was resolved when I changed to a different 
SATA controller.
My memory is a bit hazy, but I believe ZFS detected a checksum inconsistency 
during scrubbing only on the disk connected with the Marvell.
- However, since you say your USB enclosure has been in use for a long time, 
it seems the controller is not the issue.

The other case was relatively recent, and I can't go into details for some reasons, 
but there was a problem with the disk's firmware which resulted in data corruption.

Thanks,
Akagi

> 
> Thanks,
> Calvin
>