Re: 7.2-rc1 regression Folio lock leak in writepage_delalloc()

在 2026/7/22 04:41, Christian Borntraeger 写道:
> We have seen random hangs in our daily CI run where qemu/KVM
> processes  deadlocks guests with file-backed RAM on btrfs (large data folios)
> 
> With the help of claude I think we found the/one problem on an s390
> KVM host running 7.2.0-rc3 (KASAN test kernel, but the issue is not
> KASAN related). And to be honest here, most of the writeup was created
> by claude and I added things where appropriate. Also the patch was
> mostly done with the help of claude.
> 
> A KVM guest with its RAM backed by a file on btrfs (zstd compression
> enabled) locked up together with the host's writeback: two vCPU
> threads, an irqfd worker, two flusher workers, khugepaged and a
> syncfs caller (dnf) were all stuck in D state for hours.  Analysis
> of the crash dump shows a leaked folio lock in btrfs'
> writepage_delalloc(); a proposed fix is in the reply mail.
> 
> I still need to verify that this patches fixes the deadlock in our
> CI but wanted some feedback first.
> 
> Dump analysis (shortened)
> -------------------------
> All blocked tasks funnel into one 64-page (256K) large data folio of
> the guest RAM file:
> 
>    folio 0x800083fb000, inode 1881035 (the 1.25G s390.ram file)
>    flags: PG_locked | PG_waiters | PG_dirty | PG_private | PG_uptodate
>           (PG_writeback NOT set)
>    btrfs_folio_state: nr_locked == 0, subpage dirty bitmap empty
>    still mapped (63/64 PTEs) and on the LRU, no outstanding block I/O
> 
> Waiters on that folio lock:
>    - 2 vCPU threads + 1 irqfd kworker, all in
>      btrfs_page_mkwrite() -> folio_lock, holding mmap_lock (read)
> crash> bt 66448
> PID: 66448    TASK: 9e934a00          CPU: 9    COMMAND: "CPU 1/KVM"
>   #0 [b8b25dbe7d8] __schedule at c0b186d5e78
>   #1 [b8b25dbe908] schedule at c0b186d7040
>   #2 [b8b25dbe948] io_schedule at c0b186d723c
>   #3 [b8b25dbe978] folio_wait_bit_common at c0b167e719c
>   #4 [b8b25dbeaf0] btrfs_page_mkwrite at c0b172816fc
>   #5 [b8b25dbec98] do_page_mkwrite at c0b168a4ada
>   #6 [b8b25dbecf0] do_wp_page at c0b168b2350
>   #7 [b8b25dbed70] handle_pte_fault at c0b168bfaf4
>   #8 [b8b25dbee58] __handle_mm_fault at c0b168c003e
>   #9 [b8b25dbefc0] handle_mm_fault at c0b168c09b6
> #10 [b8b25dbf020] __get_user_pages at c0b16899cfc
> #11 [b8b25dbf148] get_user_pages_unlocked at c0b1689af1c
> #12 [b8b25dbf248] hva_to_pfn at c0a9711e20e [kvm]
> #13 [b8b25dbf3f0] __kvm_faultin_pfn at c0a9711ea26 [kvm]
> #14 [b8b25dbf4e8] kvm_s390_faultin_gfn at c0a971c092c [kvm]
> #15 [b8b25dbf5f8] vcpu_post_run_handle_fault at c0a97148b5e [kvm]
> #16 [b8b25dbf6f0] __vcpu_run at c0a9715c1f2 [kvm]
> #17 [b8b25dbf808] kvm_arch_vcpu_ioctl_run at c0a9715d3e4 [kvm]
> #18 [b8b25dbfbb8] kvm_vcpu_ioctl at c0a97117bd8 [kvm]
> #19 [b8b25dbfdd8] __s390x_sys_ioctl at c0b16aa3614
> #20 [b8b25dbfe40] __do_syscall at c0b186cdaee
> #21 [b8b25dbfe98] system_call at c0b186ebd42
>   USER-MODE INTERRUPT FRAME; pt_regs at b8b25dbff38:
>   PSW:  0705000180000000 000003ff8a92662c (user space)
>   GPRS: 000003ff627faf50 0000000000000036 ffffffffffffffda 000000000000ae80
>         0000000000000000 000003ff627fc8c0 000002aa1f8a7880 000003ff8a8ad310
>         000002aa1e1f3c60 0000000000000000 000000000000ae80 000002aa1f8a2f60
>         000003ff8d3adfa8 000003ff627fc8c0 000003ff627faff0 000003ff627fae88
> 
> 
>    - flusher: extent_write_cache_pages() -> folio_lock
>    - delalloc space reclaim worker: same, while holding
>      fs_info->delalloc_root_mutex (which in turn blocks
>      btrfs_async_reclaim_metadata_space on the mutex)
> Behind those: khugepaged in down_write(mmap_lock), and syncfs.
> 
> No task in the system owns the folio lock; nothing references the
> folio except the six waiters.  The lock was leaked.
> 
> Root cause
> ----------
> A folio can carry the folio-level dirty flag with an EMPTY btrfs
> subpage dirty bitmap.  btrfs data mappings use filemap_dirty_folio(),
> so a generic folio_mark_dirty() sets only the folio flag and xarray
> tag - no subpage dirty bits, no delalloc reservation.  On s390 this
> happens all the time: the KVM irq adapter path
> (arch/s390/kvm/interrupt.c, adapter_indicators_set()) pins the guest
> interrupt indicator page with pin_user_pages_remote(FOLL_WRITE),
> sets the indicator bit and calls set_page_dirty_lock().  Once a
> previously written folio has gone through one complete writeback
> cycle (subpage dirty bitmap empty again), the next adapter interrupt
> re-dirties it with only the folio flag.

No, that's not how things should work.

I have explained the problem in the RFC patch. I am only going to add 
some extra explanation inlined below.

> 
> Writeback then does:
> 
>    extent_write_cache_pages(): folio_lock(), folio is dirty -> proceed
>    extent_writepage() -> writepage_delalloc():
>      - btrfs_copy_subpage_dirty_bitmap() -> submit_bitmap is EMPTY
>      - the btrfs_folio_set_lock() loop sets nothing (nr_locked stays 0)
>      - find_lock_delalloc_range() finds nothing -> goto out
>      - out: bitmap_empty(submit_bitmap) is true -> return 1
> 
> The "return 1" path means "all dirty ranges were submitted
> asynchronously, the async submission owns the folio unlock" - but
> nothing was submitted, so extent_writepage() returns and the folio
> stays locked forever.  This matches every flag of the dump folio
> (locked, dirty, nr_locked == 0, no writeback, still mapped/LRU).
> 
> Verifying this in the dump:
> 
> - uptodate = 0xffffffffffffffff — all 64 blocks uptodate (consistent with PG_uptodate)
> - dirty = 0x0 — the subpage dirty bitmap is EMPTY, exactly as the root cause predicts
> - writeback = 0x0 — no writeback in flight (consistent with PG_writeback clear)
> 
> 
> Exposure
> --------
> - Single-block folios are immune: btrfs_copy_subpage_dirty_bitmap()
>    unconditionally reports bit 0 for blocks_per_folio == 1.

Unfortunately no.

One of the biggest problem is, all the other things, from extent map to 
ordered extent are not properly prepared.

E.g. even if btrfs_copy_subpage_dirty_bitmap() returns bit 0 set, later 
EXTENT_DELALLOC is not set.

So find_lock_delalloc_range() will return false, and since we found no 
delalloc range, @last_delalloc_end is still zero, we goto out label, 
without creating any ordered extent/extent map.

Then we go into extent_writepage_io(), which will rely on the extent_map 
created by run_delalloc_range() for IO submission.

But since we have no OE/EM created, we will grab one from on-disk 
metadata, and if the original on-disk metadata shows there is a hole, we 
will trigger the ASSERT() inside submit_one_sector(), about the EM is a 
hole.

Before v7.2-rc1, we have a lot of extra handling (folio ordered flag) to 
exactly catch such situation.
But since we haven't really hit such case anymore for a while, in 
v7.2-rc we also remove the that flag, otherwise it should catch such 
problem much earilier.

> - Subpage setups (e.g. 64K page size with 4K sectorsize) have been
>    exposed since the submission bitmap rework in v6.12
>    (bd610c0937aa "btrfs: only unlock the to-be-submitted ranges
>    inside a folio").
> - 4K page size systems became exposed with large data folio support
>    in v7.2-rc1, which routes every large folio through the subpage
>    machinery.  That is why we only started seeing this now.
> 
> Any GUP-style dirtier can trigger it (KVM adapter interrupts on
> s390, vfio, RDMA, io_uring fixed buffers, ...) as long as the target
> is a multi-block folio of a btrfs data mapping that was clean at the
> time of set_page_dirty_lock().
> 
> Reproducer outline: KVM guest on s390 with memory-backend-file on
> btrfs + virtio devices using irqfd adapter indicators;

I strongly doubt if it's a specific S390 feature breaking the assumption.

As io uring is also heavily tested, and IIRC there is already a huge GUP 
work to address the long existing unexpected dirty page behavior in v5.15.

So I strongly doubt if it's some S390x feature not properly following 
the existing scheme.

At least on both x86_64 and arm64 (64K page size), since the 
introduction of experimental large folios, I haven't seen something 
similar like this.

If it's S390X specific, then I do not think it's something we can handle 
by ourselves.

Thanks,
Qu

> hangs within
> ~25 minutes of guest uptime in our setup.  A targeted reproducer
> should also work on x86: mmap a file on btrfs, write it, fsync, let
> writeback finish, then pin_user_pages(FOLL_WRITE) +
> set_page_dirty_lock() on a page of a large folio and trigger sync.
> 
> Proposed fix
> ------------
> Detect the empty-at-entry bitmap right after it has been copied,
> before any range lock is set up, clear the stale folio dirty flag
> (nothing can ever be written back for it; all dirty flag setters
> serialize on the folio lock we hold) and unlock the folio.  Patch
> attached below; it survives our compile test and we are preparing a
> test run on the affected machine.  Comments welcome - especially on
> whether clearing the folio dirty flag is the desired semantic here,
> versus e.g. routing such folios through the cow fixup worker to
> actually persist GUP-written data.
> 
> Thanks
> 
> Christian
> 
> 
> 
> 
>