Re: [PATCH RFC 02/13] mm/huge_memory: fix rejection of swap cache folios with a mapping

On Sun, Aug 9, 2026 at 2:02 AM Zi Yan <[email protected]> wrote:
>
> On Fri Aug 7, 2026 at 5:17 PM EDT, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <[email protected]>
> >
> > A folio in the swap cache cannot be split if it has a mapping (shmem).
> > The split code only checks for this in __folio_freeze_and_split_unmapped,
> > after the folio ref has been frozen and the NR_SHMEM_THPS/NR_FILE_THPS
> > counters have been decremented, and returns -EINVAL without unfreezing
> > the folio or restoring the counters. That error path is fragile: if it
> > is ever taken, the folio is left frozen and stuck, the counters are
> > skewed, and the VM_WARN_ON_ONCE_FOLIO would fire for a state that is
> > actually legitimate.
> >
> > Check for this case up front in folio_check_splittable and return
> > -EINVAL before any state is modified. Under DEBUG_VM, the existing
> > "Tried to split an unsplittable folio" warning in __folio_split
> > reports the rejection.
>
> Should we return -EBUSY instead? -EINVAL means the caller should not
> split a swapcache shmem with a mapping and the caller needs to avoid
> that. The Fixes tag tells me a caller can split a swapcache shmem with a
> mapping, so with -EINVAL, we will want to add checks at callers to avoid
> it from happening.
>

I can drop the Fixes tags. I meant that there is already some
defensive code that trying to catch it and return -EINVAL, however,
that defensive code itself is flawed. If this situation occurs due to
a bug or future misuse, the flawed code will causes the folio to get
stuck in a frozen state. The defensive code should at least not make
things worse.

Fortunately I think no one needs to split a hybrid shmem swap cache
folio, returning -EINVAL here may help catch any potential future
misuse.