Space cache rebuild failed and reports fs corrupted
Alex Writing <[email protected]>
| Newsgroups | org.kernel.vger.linux-btrfs |
|---|---|
| Message-ID | <CAD0nGLtyhz=M-97M4YowUpGpt+b5OBhD9fJZCx3e1hU1JYXemg@mail.gmail.com> |
Hi, I just updated from Proxmox 8 to Proxmox 9, which updates from
Debian 12 to Debian 13. The new kernel is a Proxmox-specific build of
7.0.14. btrfs-progs version is the latest version Proxmox provides,
6.14.
After the update, my btrfs RAID1 HDD array failed to mount
automatically at boot because it was timing out; it now takes 90-120
seconds to mount the array for some reason. I turned on debugging and
there are lots of lines like this during the delay:
[Mon Aug 10 17:47:28 2026] BTRFS debug (device sdb1):
btrfs_submit_dev_bio: rw 0 0x1000, sector=1924895456, dev=8388625
(/dev/sdb1 id 2), size=16384
[Mon Aug 10 17:47:28 2026] BTRFS debug (device sdb1):
btrfs_submit_dev_bio: rw 0 0x1000, sector=5154454848, dev=8388625
(/dev/sdb1 id 2), size=16384
I read something online that suggested clearing the space cache. I
tried that, which ran for about 15 minutes but then failed:
[Mon Aug 10 18:12:58 2026] BTRFS: Transaction aborted (error -117)
[Mon Aug 10 18:12:58 2026] BTRFS: error (device sdb1 state A) in
__btrfs_free_extent:3419: errno=-117 Filesystem corrupted
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA): leaf
27083447287808 gen 1236591 total ptrs 184 free space 5089 owner 2
[Mon Aug 10 18:12:58 2026] item 0 key (27083429871616
METADATA_ITEM 0) itemoff 16250 itemsize 33
[Mon Aug 10 18:12:58 2026] extent refs 1 gen 1182260 flags 2
[Mon Aug 10 18:12:58 2026] ref#0: tree block backref root 2
[...]
[Mon Aug 10 18:12:58 2026] item 183 key (27083432984576
METADATA_ITEM 0) itemoff 9689 itemsize 33
[Mon Aug 10 18:12:58 2026] extent refs 1 gen 1219190 flags 258
[Mon Aug 10 18:12:58 2026] ref#0: shared block backref
parent 27105592623104
[Mon Aug 10 18:12:58 2026] BTRFS critical (device sdb1 state EA):
unable to find ref byte nr 27083430166528 parent 0 root 10 owner 1
offset 0 slot 18
[Mon Aug 10 18:12:58 2026] BTRFS error (device sdb1 state EA): failed
to run delayed ref for logical 27083430166528 num_bytes 16384 type 176
action 2 ref_mod 1: -2
[Mon Aug 10 18:12:58 2026] BTRFS: error (device sdb1 state EA) in
btrfs_run_delayed_refs:2247: errno=-2 No such entry
[Mon Aug 10 18:12:58 2026] BTRFS warning (device sdb1 state EA):
failed to rebuild free space tree: -2
[Mon Aug 10 18:12:58 2026] BTRFS error (device sdb1 state EA): commit
super block returned -30
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
space_info METADATA (sub-group id 0) has 867155968 free, is not full
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
space_info total=34359738368, used=33472987136, pinned=0,
reserved=3637248, may_use=15892480, readonly=65536 zone_unusable=0
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
global_block_rsv: size 0 reserved 0
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
trans_block_rsv: size 0 reserved 0
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
chunk_block_rsv: size 0 reserved 0
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
remap_block_rsv: size 0 reserved 0
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
delayed_block_rsv: size 0 reserved 0
[Mon Aug 10 18:12:58 2026] BTRFS info (device sdb1 state EA):
delayed_refs_rsv: size 249036800 reserved 15892480
[Mon Aug 10 18:12:58 2026] BTRFS error (device sdb1 state EA):
open_ctree failed: -2
Full output here: https://pastebin.com/aMeu1XR0
Here's what the array looks like:
Overall:
Device size: 61.85TiB
Device allocated: 33.13TiB
Device unallocated: 28.72TiB
Device missing: 0.00B
Device slack: 10.50KiB
Used: 33.12TiB
Free (estimated): 14.36TiB (min: 14.36TiB)
Free (statfs, df): 5.35TiB
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated Total Slack
-- --------- -------- -------- -------- ----------- -------- --------
2 /dev/sdb1 8.77TiB 32.00GiB 32.00MiB 3.94TiB 12.73TiB 3.50KiB
3 /dev/sdd1 11.06TiB - - 1.67TiB 12.73TiB 3.50KiB
4 /dev/sde1 11.32TiB - - 1.41TiB 12.73TiB 3.50KiB
5 /dev/sda1 1.92TiB 32.00GiB 32.00MiB 21.70TiB 23.65TiB -
-- --------- -------- -------- -------- ----------- -------- --------
Total 16.53TiB 32.00GiB 32.00MiB 28.72TiB 61.85TiB 10.50KiB
Used 16.53TiB 31.17GiB 3.38MiB
btrfs check reports "errors found in extent allocation tree or chunk
allocation" in the checking extents phase (mostly root 10 but a few in
root 1 and root 4), and in the "checking free space tree" phase
reports "could not load free space tree: No such file or directory"
which makes sense since the space tree rebuild failed. It doesn't
report any issues in any other phase.
Here's the full btrfs check output: https://pastebin.com/XZFHLLvd
The logs show that one of the drives dropped out briefly during the
btrfs check, so I'm guessing there may be a flaky cable or SATA port
that might be the root cause of the issue. I'll also probably run a
memtest before doing much more to it.
The filesystem mounts in readonly mode and all the data seems intact,
at least at first glance.
What's the best way to try to fix this? Since I can read the data I
could always copy it all off and recreate the array from scratch, but
if there's a safe way to repair things that wouldn't involve finding
that much free disk space that would be ideal.
Thanks!