Re: [PATCH RFC 09/15] mm/mglru: frequency guided workingset promotion (MGLRU-FG)

On Tue, Aug 4, 2026 at 3:47=E2=80=AFAM Kairui Song via B4 Relay
<[email protected]> wrote:
>
> From: Kairui Song <[email protected]>
>
> Complement MGLRU's eviction-time tier-PID protection with access-time
> frequency-guided promotion.  Introduce a unified set of helpers built bas=
ed
> on referenced (access) count of a folio.
>
> Each access increments a folio's referenced count stored in folio flags
> (refs), refs still mappes to a logarithmic tier just like before, but wit=
h
> more formal bit definitions, a few special thresholds are introduced:
> LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
> and LRU_REFS_MAX(7). When it reaches certain threshold, the folio is
> promoted proactively instead of wait for the PID controller to kick in.
>
> Also simplify MGLRU's usage of PG_workingset and PG_referenced, now
> these 2 flags are purely used as the lower 2 bit of refs for MGLRU. This
> doesn't effect classical LRU in any way. This will actually simplify and
> make MGLRU's certain metric reading more accurate, and reduced MGLRU's
> original tier / referenced count bit by one since only one extra bit is
> now needed to record a max referenced count of 7 (previously 2 extra bits
> are needed). This changes make sense because MGLRU doesn't have demotion
> so these 2 flags are never separately useful for MGLRU.
>
> This addresses several shortcomings of the old model:
>
> - Long feedback loop: protection only activated after enough
>   re-faults, by which time the folio is often no longer hot.
>
> - Limited tier resolution: once referenced count exceeded the bits
>   limit (8 previously), MGLRU could no longer distinguish hotter folios a=
s
>   they are capped by the tier.  And what's worse, PG_workingset
>   forces a folio to stay on tier 3.
>
> - Eviction-time bias: because PID protection activates upon eviction
>   and always targets the LRU tail, it tends to protect cold tail
>   folios at the expense of hotter head folios.  Once the tail folios
>   consume the PID protection budget, head folios lose their
>   protection.  Additionally, the PID cannot distinguish the access
>   time of folios that share the same reference count.
>
> Besides reworking the LRU_REFS related helpers and definitions, most of
> the work is done by the helpers below; the implementation details are
> described in their inline comments.
>
> - folio_inc_lru_refs(): Used by both cache access (folio_mark_accessed)
>   and page table access.  The folio could be off-list (isolated),
>   unlocked, or unmapped.  This helper uses PG_lru to stabilize the
>   folio and performs a speculative and lazy promotion.
>
> - folio_inc_lru_refs_walk(): Used by the PTE walk path during aging,
>   where generations are stable; performs lazy promotion.
>
> - folio_inc_lru_refs_isolated(): Used by the rmap check before
>   eviction.  The folio is isolated and hence this doesn't perform
>   promotion by itself; the folio will be added back to the right gen
>   upon return.
>
> The eviction-time folio_inc_gen() still handles PID protection, but the
> protection ratio is softer than before, and it caps refs at WORKINGSET
> so the folio retains enough history to stay above the cold tier.
>
> The PID controller gain factors in get_tier_idx() are also relaxed from
> (2:3) to (1:2).  Since the new folio gen bump paths already proactively
> protect hot folios, PID protection can afford to be more permissive
> without increasing the refault rate.
>
> PG_workingset and PG_referenced are repurposed as the low two bits
> of the unified LRU reference count.  LRU_REFS_MASK provides the
> higher bits.  This eliminates the old restriction where LRU_REFS_MASK
> was only valid when PG_referenced was set, and allows all paths to use
> the same encoding consistently.
>
> Hence, a workingset folio is now defined as refs >=3D LRU_REFS_WORKINGSET
> (2), matching the active/inactive LRU's definition and giving in-kernel
> consumers (PSI, readahead) consistent behavior on MGLRU, which will be
> done in later commits.
>
> Note that PG_workingset and PG_referenced are no longer independent
> flags under MGLRU.  Adjusting existing raw folio_test_*() callers
> to the new semantics is left as follow-ups.
>
> Signed-off-by: Kairui Song <[email protected]>
> ---
>  include/linux/mm_inline.h |  83 ++++++++-----
>  include/linux/mmzone.h    | 135 ++++++++++++++-------
>  kernel/bounds.c           |   2 +-
>  mm/folio.c                |  46 +-------
>  mm/vmscan.c               | 290 ++++++++++++++++++++++++++++++----------=
------
>  mm/workingset.c           |  55 ++++++---
>  6 files changed, 385 insertions(+), 226 deletions(-)
>


> +/*
> + * Update the folio's lru refs indicator without taking the folio lock,
> + * isolation, or lruvec lock. Used by both page table access (@is_fault=
=3Dtrue)
> + * and by file access (@is_fault=3Dfalse).
> + */
> +int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec)
> +{
> +       int max_gen, min_gen;
> +       int type, refs, gen, new_gen;
> +       unsigned long new_flags, old_flags, max_seq;
> +       struct lru_gen_folio *lrugen;
> +       struct lruvec *lruvec;
> +
> +       type =3D folio_is_file_lru(folio);
> +       lruvec =3D folio_lruvec_live_get(folio);
> +       lrugen =3D &lruvec->lrugen;
> +
> +       old_flags =3D READ_ONCE(*folio_flags(folio, 0));
> +       do {
> +               new_flags =3D old_flags;
> +               gen =3D lru_gen_from_flags(old_flags);
> +               refs =3D lru_refs_from_flags(old_flags) + 1;
> +               new_gen =3D gen;
> +               if (!(old_flags & BIT(PG_lru)) || gen < 0)
> +                       goto out;
> +
> +               max_seq =3D READ_ONCE(lrugen->max_seq);
> +               max_gen =3D lru_gen_from_seq(max_seq);
> +               min_gen =3D lru_gen_from_seq(READ_ONCE(lrugen->min_seq[ty=
pe]));
> +               if (gen =3D=3D max_gen)
> +                       goto out;
> +
> +               if (is_fault || is_exec) {
> +                       /* Promote second page table access or executable=
 */
> +                       if (refs > LRU_REFS_REFERENCED || is_exec)
> +                               new_gen =3D max_gen;
> +                       else
> +                               new_gen =3D (gen + 1UL) % MAX_NR_GENS;
> +                       refs =3D min(refs, LRU_REFS_PROTECTED);
> +               } else if (refs > LRU_REFS_MAX) {
> +                       /* LRU refs counting overflow, bump the gen */
> +                       new_gen =3D (gen + 1UL) % MAX_NR_GENS;
> +                       refs =3D LRU_REFS_PROTECTED;
> +               } else if (gen =3D=3D min_gen && refs >=3D LRU_REFS_WORKI=
NGSET) {
> +                       /* Defer eviction of just accessed workingset */
> +                       new_gen =3D (gen + 1UL) % MAX_NR_GENS;
> +                       refs =3D min(refs, LRU_REFS_PROTECTED);
>                 }
> +out:
> +               refs =3D min(refs, LRU_REFS_MAX);
> +               lru_refs_set_flags(&new_flags, refs);
> +               if (new_gen >=3D 0)
> +                       lru_gen_set_flags(&new_flags, new_gen);
> +       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flag=
s));
>
> -               folio_set_lru_refs(folio, 1);
> -               return false;
> +       if (new_gen !=3D gen) {
> +               /*
> +                * Gen can only go forward, so concurrent aging is
> +                * usually fine, except when multiple aging increase
> +                * max_seq multiple times, new_gen may have go beyond
> +                * the new max_seq's current gen border and causes
> +                * hotness inversion. In that very unlikely case,
> +                * just activate the folio.
> +                */
> +               lru_gen_update_size(lruvec, folio, gen, new_gen);
> +               if (unlikely(READ_ONCE(lrugen->max_seq) - max_seq > MIN_N=
R_GENS))
> +                       folio_activate(folio);
>         }

So sashiko complained about a potential race here, which I suspected
but let through for this RFC:

lru_gen_update_size is lockless here, so if the folio is move between
active and inactive gen and a concurrent aging occurs, we might lost a
active / inactive counter update and drift the reading. The chance is
tiny and active/inactive counts are just metrics reading for MGLRU so
this should have no effect on performance.

On Tue, Aug 4, 2026 at 3:47=E2=80=AFAM Kairui Song via B4 Relay
<[email protected]> wrote:
>
> From: Kairui Song <[email protected]>
>
> Complement MGLRU's eviction-time tier-PID protection with access-time
> frequency-guided promotion.  Introduce a unified set of helpers built bas=
ed
> on referenced (access) count of a folio.
>
> Each access increments a folio's referenced count stored in folio flags
> (refs), refs still mappes to a logarithmic tier just like before, but wit=
h
> more formal bit definitions, a few special thresholds are introduced:
> LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
> and LRU_REFS_MAX(7). When it reaches certain threshold, the folio is
> promoted proactively instead of wait for the PID controller to kick in.
>
> Also simplify MGLRU's usage of PG_workingset and PG_referenced, now
> these 2 flags are purely used as the lower 2 bit of refs for MGLRU. This
> doesn't effect classical LRU in any way. This will actually simplify and
> make MGLRU's certain metric reading more accurate, and reduced MGLRU's
> original tier / referenced count bit by one since only one extra bit is
> now needed to record a max referenced count of 7 (previously 2 extra bits
> are needed). This changes make sense because MGLRU doesn't have demotion
> so these 2 flags are never separately useful for MGLRU.
>
> This addresses several shortcomings of the old model:
>
> - Long feedback loop: protection only activated after enough
>   re-faults, by which time the folio is often no longer hot.
>
> - Limited tier resolution: once referenced count exceeded the bits
>   limit (8 previously), MGLRU could no longer distinguish hotter folios a=
s
>   they are capped by the tier.  And what's worse, PG_workingset
>   forces a folio to stay on tier 3.
>
> - Eviction-time bias: because PID protection activates upon eviction
>   and always targets the LRU tail, it tends to protect cold tail
>   folios at the expense of hotter head folios.  Once the tail folios
>   consume the PID protection budget, head folios lose their
>   protection.  Additionally, the PID cannot distinguish the access
>   time of folios that share the same reference count.
>
> Besides reworking the LRU_REFS related helpers and definitions, most of
> the work is done by the helpers below; the implementation details are
> described in their inline comments.
>
> - folio_inc_lru_refs(): Used by both cache access (folio_mark_accessed)
>   and page table access.  The folio could be off-list (isolated),
>   unlocked, or unmapped.  This helper uses PG_lru to stabilize the
>   folio and performs a speculative and lazy promotion.
>
> - folio_inc_lru_refs_walk(): Used by the PTE walk path during aging,
>   where generations are stable; performs lazy promotion.
>
> - folio_inc_lru_refs_isolated(): Used by the rmap check before
>   eviction.  The folio is isolated and hence this doesn't perform
>   promotion by itself; the folio will be added back to the right gen
>   upon return.
>
> The eviction-time folio_inc_gen() still handles PID protection, but the
> protection ratio is softer than before, and it caps refs at WORKINGSET
> so the folio retains enough history to stay above the cold tier.
>
> The PID controller gain factors in get_tier_idx() are also relaxed from
> (2:3) to (1:2).  Since the new folio gen bump paths already proactively
> protect hot folios, PID protection can afford to be more permissive
> without increasing the refault rate.
>
> PG_workingset and PG_referenced are repurposed as the low two bits
> of the unified LRU reference count.  LRU_REFS_MASK provides the
> higher bits.  This eliminates the old restriction where LRU_REFS_MASK
> was only valid when PG_referenced was set, and allows all paths to use
> the same encoding consistently.
>
> Hence, a workingset folio is now defined as refs >=3D LRU_REFS_WORKINGSET
> (2), matching the active/inactive LRU's definition and giving in-kernel
> consumers (PSI, readahead) consistent behavior on MGLRU, which will be
> done in later commits.
>
> Note that PG_workingset and PG_referenced are no longer independent
> flags under MGLRU.  Adjusting existing raw folio_test_*() callers
> to the new semantics is left as follow-ups.
>
> Signed-off-by: Kairui Song <[email protected]>
> ---
>  include/linux/mm_inline.h |  83 ++++++++-----
>  include/linux/mmzone.h    | 135 ++++++++++++++-------
>  kernel/bounds.c           |   2 +-
>  mm/folio.c                |  46 +-------
>  mm/vmscan.c               | 290 ++++++++++++++++++++++++++++++----------=
------
>  mm/workingset.c           |  55 ++++++---
>  6 files changed, 385 insertions(+), 226 deletions(-)
>


> +/*
> + * Update the folio's lru refs indicator without taking the folio lock,
> + * isolation, or lruvec lock. Used by both page table access (@is_fault=
=3Dtrue)
> + * and by file access (@is_fault=3Dfalse).
> + */
> +int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec)
> +{
> +       int max_gen, min_gen;
> +       int type, refs, gen, new_gen;
> +       unsigned long new_flags, old_flags, max_seq;
> +       struct lru_gen_folio *lrugen;
> +       struct lruvec *lruvec;
> +
> +       type =3D folio_is_file_lru(folio);
> +       lruvec =3D folio_lruvec_live_get(folio);
> +       lrugen =3D &lruvec->lrugen;
> +
> +       old_flags =3D READ_ONCE(*folio_flags(folio, 0));
> +       do {
> +               new_flags =3D old_flags;
> +               gen =3D lru_gen_from_flags(old_flags);
> +               refs =3D lru_refs_from_flags(old_flags) + 1;
> +               new_gen =3D gen;
> +               if (!(old_flags & BIT(PG_lru)) || gen < 0)
> +                       goto out;
> +
> +               max_seq =3D READ_ONCE(lrugen->max_seq);
> +               max_gen =3D lru_gen_from_seq(max_seq);
> +               min_gen =3D lru_gen_from_seq(READ_ONCE(lrugen->min_seq[ty=
pe]));
> +               if (gen =3D=3D max_gen)
> +                       goto out;
> +
> +               if (is_fault || is_exec) {
> +                       /* Promote second page table access or executable=
 */
> +                       if (refs > LRU_REFS_REFERENCED || is_exec)
> +                               new_gen =3D max_gen;
> +                       else
> +                               new_gen =3D (gen + 1UL) % MAX_NR_GENS;
> +                       refs =3D min(refs, LRU_REFS_PROTECTED);
> +               } else if (refs > LRU_REFS_MAX) {
> +                       /* LRU refs counting overflow, bump the gen */
> +                       new_gen =3D (gen + 1UL) % MAX_NR_GENS;
> +                       refs =3D LRU_REFS_PROTECTED;
> +               } else if (gen =3D=3D min_gen && refs >=3D LRU_REFS_WORKI=
NGSET) {
> +                       /* Defer eviction of just accessed workingset */
> +                       new_gen =3D (gen + 1UL) % MAX_NR_GENS;
> +                       refs =3D min(refs, LRU_REFS_PROTECTED);
>                 }
> +out:
> +               refs =3D min(refs, LRU_REFS_MAX);
> +               lru_refs_set_flags(&new_flags, refs);
> +               if (new_gen >=3D 0)
> +                       lru_gen_set_flags(&new_flags, new_gen);
> +       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flag=
s));
>
> -               folio_set_lru_refs(folio, 1);
> -               return false;
> +       if (new_gen !=3D gen) {
> +               /*
> +                * Gen can only go forward, so concurrent aging is
> +                * usually fine, except when multiple aging increase
> +                * max_seq multiple times, new_gen may have go beyond
> +                * the new max_seq's current gen border and causes
> +                * hotness inversion. In that very unlikely case,
> +                * just activate the folio.
> +                */
> +               lru_gen_update_size(lruvec, folio, gen, new_gen);
> +               if (unlikely(READ_ONCE(lrugen->max_seq) - max_seq > MIN_N=
R_GENS))
> +                       folio_activate(folio);
>         }

So sashiko complained about a potential race here, which I suspected
but let through for this RFC:

lru_gen_update_size is lockless here, so if the folio is move between
active and inactive gen and a concurrent aging occurs, we might lost a
active / inactive counter update and drift the reading. The chance is
tiny and active/inactive counts are just metrics reading for MGLRU so
this should have no effect on performance.

I'll need to figure out a way to fix this, hopefully without any lock
because this is a hot path where overheads are critical.

One idea is that we directly jump to the part where we decouple the
active/inactive counter with MGLRU's gens (I mentiones this before
just didn't post any code about it yet)? If you use MGLRU in a fleet
you should already have noticed that the active / inactive reading is
almost unusable, because MGLRU's aging moves folios in whole
generations. Because the active/inactive count is bounded by
generations, you will see the numbers are very "jumpy," and almost all
page cache is always reported as inactive since the PID can only
promote the page cache to the second oldest generation, which is
considered "inactive." This is incorrect and breaks things like
cAdvisor in K8s, which uses the inactive reading to determine how much
of the page cache is actually used.

So instead, if we bound active / inactive with tiers instead of gens,
all the issues are gone. I planned to do that later, but if everyone
is OK with it, we can do this earlier.

Or else, some fancy atomic / barrier might be needed, I think that's
still doable though but kind of complex and maybe ugly.