[PATCH 0/9] mm/mglru: suppress empty page table walks during aging

In the current MGLRU, lru_gen_use_mm() will mark an mm for all nodes at
each context switch, so on each node aging will walk into each mm's page
table independently. On a multi-NUMA node system, one process launched
on one or a subset of nodes, its mm is walked by all other nodes's aging
while finds on pages for their lruvec. This is pure waste (100% walks
are empty on those other nodes)

This patch series suppresses these empty page table walks with two
complementary mechanisms at different levels:

  - empty_map (cross-node specifc). At mm granularity, one process's mm
    whose walk found no pages for one node's lruvec is skipped on that
    node between re-scan passes.

  - PUD-level Bloom filter (general). One level up from the existing PMD
    filters, it skips any 1GB PUD that had no young entries last
    generation - namely whose 512 PMDs all failed PMD test. Cross-node
    empty walks can prove its effect the best, but it also suppresses
    purely code PUD inside local mms. So it reduces unnecessary walking
    in any workload with cold areas.

The two complement each other: empty_map reduces the *number* of
cross-node walks, the PUD-level filter reduces the *cost* of the walks
that remain.

Measurement test (3-socket Xeon 6766E, 32 x 256MB workers pinned to node 0):

  node 1 (foreign mm walks):
    empty_map off (PUD filter on): 3300 walks, all empty, ~1238 PTE scans/walk
    both on:                       900 walks (-73%), ~0 PTE scans/walk

  - empty_map cuts node 1's walk count by ~73% (every 4th pass re-scans
    to close migration/fault windows).
  - the PUD-level filter makes the remaining walks nearly free: the 1GB
    subtrees are skipped instead of iterating their 512 PMDs
    (leaf_total drops from ~1238 to ~0 per walk).
  - worker RSS is unchanged in both modes - no premature reclaim. A
    force_scan pass confirms the local pages are found young and eligible
    (leaf_eligible == young == 21320 on node 0) when the filters are
    bypassed, i.e. the pages are hot and the aging walker can find them.

Performance regression test (make -j4 in a 3G memory cgroup, 4 vCPU / 8GB
2-NUMA VM, median of 3 runs):

                  baseline   patch    diff
    build time     10m01s    9m41s    -3.4%
    pgpgin        136868    138080   +0.9%
    pgmajfault      1832      1862   +1.6%

Build time, page-in and major-fault counts are within run-to-run
variance of a no-patch baseline - no measurable regression. The build
time is if anything slightly lower, consistent with the suppression
reducing reclaim overhead (fewer cross-node empty walks) during a real
memory-pressure workload.

The test codes/scripts (numa_workload + run_test_v4.sh) are available at:
https://github.com/baoquan-he/mglru-empty-walk-test

RFC-v1:
  - Bloom filter helpers are named symmetrically -
    test/update/reset_pmd_bloom_filter() alongside the PUD-level ones, and
    the struct field filters -> pmd_filters (Barry).

  - An mm is marked empty only when its page tables were actually walked
    (a failed mmap_read_trylock() or a stale seq is not empty), and the skip
    is invalidated on the major page-fault and migration paths. (Sashiko)

  - Reworked the re-scan: the old shared counter across nodes got stuck at
    the slowest node, so the skip never really engaged. Now each node re-scans
    its empty mms every N passes on its own clock. N is the skip_empty knob
    (default 4, proper read/write with input validation). (Sashiko)

  - Added measurement counters/tracepoint, and a kernel-build regression
    test (no measurable impact).


Baoquan He (9):
  mm/mglru: add MM_WALK_EMPTY stats and tracepoint
  mm/mglru: suppress cross-node empty page table walks
  mm/mglru: add debugfs knob for the empty-walk skip threshold
  mm/mglru: invalidate empty-walk skip on page fault and migration
  mm/mglru: add PUD-level Bloom filter state
  mm/mglru: refactor Bloom filter helpers for two filter levels
  mm/mglru: skip PUD subtrees during aging
  mm/mglru: report hot PUDs from the rmap feedback path
  mm/mglru: count PUD subtrees skipped by the PUD-level filter

 include/linux/mm_types.h      |  22 ++++
 include/linux/mmzone.h        |  13 ++-
 include/trace/events/vmscan.h |  30 +++++
 mm/huge_memory.c              |   3 +
 mm/memory.c                   |  15 +++
 mm/migrate.c                  |   4 +
 mm/vmscan.c                   | 205 +++++++++++++++++++++++++++++-----
 7 files changed, 264 insertions(+), 28 deletions(-)


base-commit: efecab401cb15fd3bb9bc05990609acb6b267ff2
-- 
2.54.0