Re: [REGRESSION] Silent SATA read corruption with dma-iommu on AMD 600-series AHCI (6.19 good, 7.0+ bad)

Mikael,

Thanks for the detailed report!

Can you please apply Commit 1e75a8255f11c81fb and retest. Also boot with
`amd_iommu=pgtbl_v2`?


@Mario, Can you look into this issue? Is this similar to the one you were
debugging internally?


-Vasant


On 8/28/2026 10:20 AM, Mikael Etienne wrote:
> You don't often get email from [email protected]. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
> Hi,
> 
> Since Linux 7.0 I observe silent read corruption on SATA disks behind an AMD
> 600-series chipset AHCI controller. AHCI/libata, the block layer, the IOMMU and
> PCIe AER report no error at all; only consumers that validate the returned bytes
> (btrfs data checksums, fio --verify) notice. SMART stays PASSED throughout.
> 
> Hardware, up front because it may matter: AMD Ryzen 7 8700G, with the SATA
> controller integrated in the chipset:
> 
> 0e:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD]
> 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
> 
> This is the same SoC as bug 219609, where dma-iommu was already identified as the
> trigger for a different device class (NVMe). Details in section 6.
> 
> Two observations sharply constrain the failure:
> 
> 1. Once corruption starts, rebooting with the same kernel and the same command
> line, WITHOUT rewriting the test file, makes the very same blocks verify
> correctly again. Continued reads can later trigger the failure again. I have
> therefore observed no persistent corruption of the test file contents; this
> behaves like transient read-path state.
> 
> 2. Adding only "iommu=pt" to the command line (nothing else changed): no
> corruption observed over 40 consecutive verification passes, 10 TiB re-read
> in 17 h. In the default translated mode the same test failed after 768 GiB
> / 3 h 25 m.
> 
> Good/bad boundary: 6.18.16 and 6.19.x are clean, every 7.0.x and 7.1.x is
> affected. Kernel is untainted (/proc/sys/kernel/tainted = 0), no out-of-tree or
> DKMS modules loaded.
> 
> Important caveat, stated up front: all kernels tested so far are Fedora-packaged.
> I have not confirmed this on vanilla upstream and I have not bisected. See "What
> I can and cannot do" at the end.
> 
> #regzbot introduced: v6.19..v7.0
> 
> 
> ## 1. Test matrix
> 
> Test file: 256 GiB fio canary with crc32c verification, on btrfs, on the 12 TB
> drive (WDC WD120EFGX-68CPHN0).
> 
> 7.1.10-200.fc44, translated (DMA-FQ):
> first verify failure after 768 GiB re-read / 3 h 25 m
> once in the failed state, a plain 4 GiB O_DIRECT read of the same file
> produced tens of thousands of further csum failures
> btrfs corruption_errs reached 20,858,094 in that session
> reboot -> counters back to 0, same file verifies clean again
> 
> 7.1.10-200.fc44, iommu=pt (identity):
> 40 consecutive clean passes, 10 TiB re-read, 17 h
> btrfs corruption_errs = 0, zero kernel error lines
> 
> Earlier, translated mode, other workloads on the 8 TB drive:
> read-only workload 9 h 30 m clean
> deduplication workload failed at 3 h 17 m
> sequential write failed at 6 h 35 m
> 
> I have not yet run enough boots to give a proper min/median/max distribution.
> The 3 h 25 m figure above is a single, carefully instrumented data point.
> 
> 
> ## 2. Reproducer
> 
> fio 3.40, io_uring engine (libaio not tested yet).
> 
> Write the canary once:
> 
> fio --name=canary --filename=/srv/12to/.sata-canary --size=256G --bs=128k \
> --ioengine=io_uring --direct=1 --iodepth=32 \
> --verify=crc32c --verify_interval=4096 --rw=write \
> --do_verify=0 --fsync_on_close=1
> 
> Then loop the verification until it fails:
> 
> while :; do
> fio --name=canary --filename=/srv/12to/.sata-canary --size=256G --bs=128k \
> --ioengine=io_uring --direct=1 --iodepth=32 \
> --verify=crc32c --verify_interval=4096 --rw=write \
> --verify_only=1 --verify_fatal=1 || break
> done
> 
> The reboot-without-rewrite protocol: when it fails, reboot and re-run only the
> verification loop above. The canary file is never rewritten. It verifies clean.
> 
> 
> ## 3. Corruption signature
> 
> The frequent mode is a 4096-byte page read back as all zeros. btrfs reports
> "csum 0x8941f998", which is the CRC32C of a zero-filled 4 KiB block.
> 
> The interesting mode is a structured permutation. From one btrfs report, inode
> 6646:
> 
> offset A offset B delta
> 1314816 1355776 40960
> 1318912 1351680 32768
> 1323008 1347584 24576
> 1327104 1343488 16384
> 1331200 1339392 8192
> 
> Verified programmatically from the raw btrfs csum lines: the data read at offset
> A is exactly what was expected at offset B, and vice versa. Five reciprocal
> pairs, symmetric around the pivot at 1335296 -- a run of 11 consecutive 4 KiB
> blocks in reverse order, the centre block mapping onto itself.
> 
> This pattern suggests incorrect DMA/scatterlist mapping or descriptor handling,
> because the affected chunks form a structured permutation rather than random bit
> corruption. It does not by itself identify the faulty layer.
> 
> 
> ## 4. Hardware and storage stack
> 
> Gigabyte X870I AORUS PRO ICE, BIOS FB1c (2026-07-21)
> AMD Ryzen 7 8700G w/ Radeon 780M Graphics
> 32 GB DDR5 non-ECC, single stick, JEDEC 4800 (EXPO/XMP off)
> 
> 0e:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD]
> 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
> 
> ahci flags: 64bit ncq sntf stag pm led clo only pmp pio slum part sxs deso
> sadm sds apst
> 32 command slots, queue_depth 32, max_segment_size 65536, max_segments 168,
> max_sectors_kb 4096, scheduler bfq
> 
> IOMMU: AMD-Vi, "Default domain type: Translated",
> "DMA domain TLB invalidation policy: lazy mode"
> Controller iommu_group type: DMA-FQ, becomes "identity" with iommu=pt
> 
> Test drive: /dev/sdb1 -> /srv/12to
> btrfs, data single, metadata DUP, mounted rw,noatime,compress=zstd:3,
> space_cache=v2. Plain partition: no LVM, no dm-crypt, no mdraid.
> 
> Drives that showed the symptom (three different drives, one taken new out of
> its box and affected within hours of first use):
> WDC WD120EFGX-68CPHN0, fw 85.00B85 (btrfs, quantified above)
> Seagate ST8000VN004-2M2101, fw SC60 (btrfs)
> WDC WD101EFBX-68B0AN0 (ext4)
> 
> NVMe devices in the same machine have never shown the symptom.
> 
> 
> ## 5. What I ruled out
> 
> - NCQ: queue_depth 32 -> 1, no change (304 vs 769 csum failures per 4 GiB read)
> - transfer size: max_sectors_kb 4096 -> 64, no change (388 vs 342)
> - both combined: no change
> - SATA link power management: already max_performance on the affected ports
> - PCIe ASPM: disabled on that link
> - PCIe AER: all correctable and non-fatal counters at 0
> - SATA link CRC (SMART attribute 199): 0 on every drive
> - temperature: 41-56 C
> - swiotlb=force does NOT help, which is consistent with the dma-iommu path
> still being used underneath
> - not btrfs-specific: at the same moment, ext4 on a second drive reported
> "bad header/extent: invalid magic - magic 0", and parted reported a corrupt
> GPT on it. Both drives read correctly again after a reboot.
> 
> btrfs is what first exposed the problem, through its data checksums. Filesystems
> without user-data checksums may hand affected data to userspace without noticing,
> although metadata validation or application-level checksums can still catch some
> of it -- which is what happened with ext4 and parted above.
> 
> 
> ## 6. Possibly related
> 
> Bug 219609, "File corruptions on SSD in 1st M.2 socket of AsRock X600M-STX +
> Ryzen 8700G". Christoph Hellwig writes there that "the problem only happens when
> using the dma-iommu code (with or without swiotlb buffering for unaligned /
> untrusted data)", and that iommu=pt or amd_iommu=off fix it.
> 
> Same SoC (Ryzen 8700G) as this machine, but a different device class (SATA/AHCI
> here, NVMe there) and a different kernel window, so I am reporting separately and
> cross-referencing rather than piling onto that bug.
> 
> 
> ## 7. What I can and cannot do
> 
> This is a production home server, not a test bench, so I want to be straight
> about it:
> 
> I CAN: run any specific test, boot parameter or debug patch you ask for, and
> report back with full instrumentation. I have a working reproducer and
> a drive I can dedicate to it.
> 
> I CANNOT realistically: dedicate the machine to a multi-day v6.19..v7.0
> bisection. Classifying a kernel as "good" currently costs several hours
> and several TiB of reads, which makes ~13 bisection steps impractical
> for me. If someone can suggest a faster trigger, that changes.
> 
> I have not yet tested: vanilla upstream kernels, libaio instead of io_uring,
> iommu.strict=1, or amd_iommu=off. I am happy to test any of these.
> 
> Available on request, immediately: full dmesg from a bad boot and from an
> iommu=pt boot, kernel .config for good and bad, /proc/cmdline, uname -a,
> /proc/sys/kernel/tainted, lspci -nnvv, lspci -t, IOMMU group and domain types,
> queue/DMA sysfs attributes, SMART reports, AER counters, raw fio logs, the raw
> btrfs csum lines, and the script that proved the A/B swaps.
> 
> Thanks,
>