Null value in ZFS linked list
John F Carr <[email protected]>
| Newsgroups | gmane.os.freebsd.devel.file-systems |
|---|---|
| Message-ID | <[email protected]> |
I have repeated panics in ZFS because a vdev_queue_t vq_active_list linked list has a NULL pointer. Any idea how to debug this?
In one recent crash, list_remove assert failed because the next pointer is null.
In another, list_remove crashed because the prev pointer is null.
Both stack traces look the same
list_remove
vdev_queue_pending_remove
vdev_queue_io_done
zio_vdev_io_done
__zio_execute
zio_execute
taskqueue_run_locked
These two crashes are on stable/14 1cbd6e148249 "libnv: fix heap overflow in nvlist_recv()". I had an identical crash involving a different pool on the same machine on 14.2. See bug 285162.
My other previous reported ZFS crashes: 276420, 277267 (same machine, different pool, panic due to slow I/O or I/O error), 268909 (raidz2 pool on another machine), 262421 (unexplained checksum errors). All have spinning disks, HPE hardware (2 kinds), and AMD processors (2 kinds).
The most recent crashes are due to a problematic pool, raidz2 on five spinning disks, on a HPE Proliant DL325. I ruled out the disk controller as a problem by moving the disks to a USB enclosure. My understanding is the vdev_queue_t structure is only in memory so the disk and controller shouldn't matter, except to the extent that disk performance affects race conditions.
NAME SIZE ALLOC FREE AG CAP DEDUP HEALTH ALTROOT
private 22.6T 17.4T 5.18T 3% 77% 1.15x DEGRADED -
raidz2-0 22.6T 17.4T 5.18T 3% 77.1% - DEGRADED
da4 4.55T - - - - - ONLINE
replacing-1 - - - - - - DEGRADED
da3 - - - - - - OFFLINE
diskid/DISK-525500000A01 4.55T - - - - - ONLINE
diskid/DISK-525500000A03 4.55T - - - - - ONLINE
da6 4.55T - - - - - ONLINE
diskid/DISK-525500000A05 4.55T - - - - - ONLINE
cache - - - - - - -
nda0p4 80G 79.8G 205M 0% 99.7% - ONLINE
pool: private
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Apr 27 13:26:42 2026
12.0T / 17.4T scanned at 1.53G/s, 520G / 17.4T issued at 66.5M/s
104G resilvered, 2.91% done, 3 days 02:12:42 to go
config:
NAME STATE READ WRITE CKSUM
private DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
da4 ONLINE 0 0 0
replacing-1 DEGRADED 0 0 0
da3 OFFLINE 0 0 0
diskid/DISK-525500000A01 ONLINE 0 0 0 (resilvering)
diskid/DISK-525500000A03 ONLINE 0 0 0
da6 ONLINE 0 0 0
diskid/DISK-525500000A05 ONLINE 0 0 0
cache
nda0p4 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
<metadata>:<0xe95>
<metadata>:<0x20e4>
(pathname redacted):<0x0>
John Carr