[to-be-updated] mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios.patch removed from -mm tree
Andrew Morton <[email protected]>
| Newsgroups | org.kernel.vger.mm-commits |
|---|---|
| Message-ID | <[email protected]> |
The quilt patch titled
Subject: mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios
has been removed from the -mm tree. Its filename was
mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios.patch
This patch was dropped because an updated version will be issued
------------------------------------------------------
From: "Lorenzo Stoakes (ARM)" <[email protected]>
Subject: mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios
Date: Thu, 06 Aug 2026 21:21:39 +0100
Currently anonymous folios belonging to CoW'd MAP_PRIVATE file-backed
mappings are indexed by their page offset within the file in which they
were originally mapped.
This differs from anonymous folios belonging to pure anon mappings which
are indexed by their anonymous page offset (the address at which they'd
belong in the VMA when first faulted).
This change fixes this inconsistency, always indexing anonymous folios by
their anonymous page offset regardless of the VMA to which they belong.
The foundations have been laid such that we need only switch this
functionality on such by:
* Using linear_anon_page_index() in __folio_set_anon() to assign the
folio's index to the anonymous linear index rather than the file-backed
one.
* Otherwise using linear_anon_page_index() in all instances where
anonymous folios are being referenced or manipulated.
* Replacing vma_address() with vma_filebacked_address() or
vma_anon_address() as appropriate.
* Updating the merging logic to check that anonymous page offsets are
aligned as well as filebacked ones for MAP_PRIVATE file-backed VMAs,
introducing needs_adjacent_anon_pgoff() to figure out when this is
required.
* Updating linear_folio_page_index() to invoke linear_anon_page_index()
if the folio is anonymous.
* Updating vma_address_end() to use the VMA's anonymous page offset when
pvmw->pgoff is anonymous.
* Correcting folio_within_range() to use anonymous page offset for
anonymous folios.
This will have no impact on merging of anonymous VMAs, whose page offset
and anonymous page offset are identical, nor will it impact shared
file-backed VMAs, which will continue to be merged based on the
file-backed page offset.
However, MAP_PRIVATE file-backed mappings must now be aligned on anonymous
page offset as well.
In most instances this should have no impact on merging of file-backed
mappings, which are usually not merged all that often, let alone
MAP_PRIVATE mapped ones, and rarely remapped and faulted before being
moved back in place (the case in which a merge may now fail).
One subtle impact of this change is in NUMA interleaving - since commit
88c91dc58582 ("mempolicy: migration attempt to match interleave nodes"),
migration heuristically tries to maintain interleaving behaviour matching
the policy using folio indices.
When doing migration of CoW'd MAP_PRIVATE-file backed ranges, the 'base'
upon which the interleaving behaviour is performed will vary for these
ranges. However the commit notes that ranges spanning multiple VMAs will
already cause varying bases, and that this is an acceptable approximation.
It is very unlikely real world use-cases will be impacted by this
(MAP_PRIVATE file-backed mappings are already an edge case), and all that
will happen is that such ranges will cause interleaving to be rotated over
the CoW'd range, with little to no impact.
This commit lays the foundations for future scalable CoW work which needs
to track some remaps, meaning that most remap tracking can be avoided, and
in nearly all cases the anonymous page offset will be able to be used to
quickly find the VMA in an mm.
Note that the need_rmap_locks check doesn't need to be updated, as any
remapping will offset both the anonymous and file-backed page offset, so
it suffices to check only one.
Link: https://lore.kernel.org/20260806-b4-scalable-cow-virt-pgoff-v4-14-ab318a350404@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <[email protected]>
Acked-by: David Hildenbrand (Arm) <[email protected]>
Cc: Adrian Hunter <[email protected]>
Cc: Alexander Deucher <[email protected]>
Cc: Alexander Gordeev <[email protected]>
Cc: Alexander Shishkin <[email protected]>
Cc: Alistair Popple <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Baolin Wang <[email protected]>
Cc: Baoquan He <[email protected]>
Cc: Barry Song <[email protected]>
Cc: Boris Brezillon <[email protected]>
Cc: Byungchul Park <[email protected]>
Cc: Chengming Zhou <[email protected]>
Cc: Chris Li <[email protected]>
Cc: Christan König <[email protected]>
Cc: Christian Borntraeger <[email protected]>
Cc: Claudio Imbrenda <[email protected]>
Cc: Dave Airlie <[email protected]>
Cc: Dev Jain <[email protected]>
Cc: Gerald Schaefer <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Gregory Price (Meta) <[email protected]>
Cc: Harry Yoo <[email protected]>
Cc: Heiko Carstens <[email protected]>
Cc: Huang Ray <[email protected]>
Cc: "Huang, Ying" <[email protected]>
Cc: Ian Rogers <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: James Clark <[email protected]>
Cc: Jan Kara <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Janosch Frank <[email protected]>
Cc: Jason Gunthorpe <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: John Hubbard <[email protected]>
Cc: Joshua Hahn <[email protected]>
Cc: Kairui Song <[email protected]>
Cc: Kees Cook <[email protected]>
Cc: Kemeng Shi <[email protected]>
Cc: Lance Yang <[email protected]>
Cc: Liam R. Howlett <[email protected]>
Cc: Liviu Dudau <[email protected]>
Cc: Maarten Lankhorst <[email protected]>
Cc: Marc Rutland <[email protected]>
Cc: "Masami Hiramatsu (Google)" <[email protected]>
Cc: Matthew Auld <[email protected]>
Cc: Matthew Brost <[email protected]>
Cc: Matthew Wilcox (Oracle) <[email protected]>
Cc: Maxime Ripard <[email protected]>
Cc: Miaohe Lin <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Mike Rapoport <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Namhyung kim <[email protected]>
Cc: Naoya Horiguchi <[email protected]>
Cc: Nhat Pham <[email protected]>
Cc: Nico Pache <[email protected]>
Cc: Oleg Nesterov <[email protected]>
Cc: Oscar Salvador <[email protected]>
Cc: Pedro Falcato <[email protected]>
Cc: Peter Xu <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Rakie Kim <[email protected]>
Cc: Rik van Riel <[email protected]>
Cc: Rodrigo Vivi <[email protected]>
Cc: Ryan Roberts <[email protected]>
Cc: Steven Price <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Sven Schnelle <[email protected]>
Cc: <[email protected]>
Cc: Thomas Hellström <[email protected]>
Cc: Thomas Zimemrmann <[email protected]>
Cc: Vasily Gorbik <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Xu Xin <[email protected]>
Cc: Zi Yan <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
---
mm/huge_memory.c | 2 +-
mm/internal.h | 27 ++++++++-------------------
mm/interval_tree.c | 4 ++--
mm/ksm.c | 6 +++---
mm/page_vma_mapped.c | 2 +-
mm/rmap.c | 12 ++++++------
mm/userfaultfd.c | 4 ++--
mm/vma.c | 32 +++++++++++++++++++++++++++++++-
8 files changed, 54 insertions(+), 35 deletions(-)
--- a/mm/huge_memory.c~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/huge_memory.c
@@ -2930,7 +2930,7 @@ int move_pages_huge_pmd(struct mm_struct
}
folio_move_anon_rmap(src_folio, dst_vma);
- src_folio->index = linear_page_index(dst_vma, dst_addr);
+ src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
_dst_pmd = folio_mk_pmd(src_folio, dst_vma->vm_page_prot);
/* Follow mremap() behavior and treat the entry dirty after the move */
--- a/mm/internal.h~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/internal.h
@@ -945,7 +945,8 @@ folio_within_range(struct folio *folio,
return false;
pgoff_folio = folio_pgoff(folio);
- pgoff_vma_start = vma_start_pgoff(vma);
+ pgoff_vma_start = folio_test_anon(folio) ?
+ vma_start_anon_pgoff(vma) : vma_start_pgoff(vma);
if (start < vma->vm_start)
start = vma->vm_start;
@@ -1056,23 +1057,8 @@ static inline unsigned long vma_fileback
}
/**
- * vma_address - Find the virtual address a page range is mapped at.
- * @vma: The vma which maps this object.
- * @pgoff: The page offset within its object.
- * @nr_pages: The number of pages to consider.
- *
- * If any page in this range is mapped by this VMA, return the first address
- * where any of these pages appear. Otherwise, return -EFAULT.
- */
-static inline unsigned long vma_address(const struct vm_area_struct *vma,
- pgoff_t pgoff, unsigned long nr_pages)
-{
- return __vma_address(vma, pgoff, vma_start_pgoff(vma), nr_pages);
-}
-
-/**
- * vma_anon_address - Find the address an anonymous folio with index @pgoff_anon
- * is mapped at.
+ * vma_anon_address - Find the virtual address an anonymous page range is mapped
+ * at.
* @vma: The vma which maps this object.
* @pgoff_anon: The anonymous page index belonging to the folio.
* @nr_pages: The number of pages to consider.
@@ -1105,7 +1091,10 @@ static inline unsigned long vma_address_
if (pvmw->nr_pages == 1)
return pvmw->address + PAGE_SIZE;
- pgoff_vma_start = vma_start_pgoff(vma);
+ if (pvmw->pgoff_is_anon)
+ pgoff_vma_start = vma_start_anon_pgoff(vma);
+ else
+ pgoff_vma_start = vma_start_pgoff(vma);
address = vma->vm_start +
((pgoff_end - pgoff_vma_start) << PAGE_SHIFT);
--- a/mm/interval_tree.c~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/interval_tree.c
@@ -83,12 +83,12 @@ mapping_rmap_tree_iter_next(struct vm_ar
static pgoff_t avc_start_pgoff(struct anon_vma_chain *avc)
{
- return vma_start_pgoff(avc->vma);
+ return vma_start_anon_pgoff(avc->vma);
}
static pgoff_t avc_last_pgoff(struct anon_vma_chain *avc)
{
- return vma_last_pgoff(avc->vma);
+ return vma_last_anon_pgoff(avc->vma);
}
INTERVAL_TREE_DEFINE(struct anon_vma_chain, rb, pgoff_t, rb_subtree_last,
--- a/mm/ksm.c~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/ksm.c
@@ -1625,7 +1625,7 @@ static int try_to_merge_with_ksm_page(st
* stable_tree, break_cow() will clean it up.
*/
rmap_item->anon_vma = vma->anon_vma;
- rmap_item->linear_page_index = linear_page_index(vma, rmap_item->address);
+ rmap_item->linear_page_index = linear_anon_page_index(vma, rmap_item->address);
get_anon_vma(vma->anon_vma);
out:
mmap_read_unlock(mm);
@@ -3152,7 +3152,7 @@ struct folio *ksm_might_need_to_copy(str
return folio; /* no need to copy it */
} else if (!anon_vma) {
return folio; /* no need to copy it */
- } else if (folio->index == linear_page_index(vma, addr) &&
+ } else if (folio->index == linear_anon_page_index(vma, addr) &&
anon_vma->root == vma->anon_vma->root) {
return folio; /* still no need to copy it */
}
@@ -3222,7 +3222,7 @@ again:
/*
* Currently, KSM folios are always small folios, so it's
* sufficient to search for a single page. We can simply use
- * the linear_page_index of the original de-duplicate
+ * the linear_anon_page_index of the original de-duplicate
* anonymous page that we remembered in the rmap_item while
* de-duplicating. Note that mremap() always de-duplicates KSM
* folios: so if there was mremap() in our parent or our child,
--- a/mm/page_vma_mapped.c~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/page_vma_mapped.c
@@ -365,7 +365,7 @@ unsigned long page_mapped_in_vma(const s
};
if (folio_test_anon(folio))
- pvmw.address = vma_address(vma, pgoff, 1);
+ pvmw.address = vma_anon_address(vma, pgoff, 1);
else
pvmw.address = vma_filebacked_address(vma, pgoff, 1);
if (pvmw.address == -EFAULT)
--- a/mm/rmap.c~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/rmap.c
@@ -866,7 +866,7 @@ unsigned long page_address_in_vma(const
vma->anon_vma->root != anon_vma->root)
return -EFAULT;
/* KSM folios don't reach here because of the !anon_vma check */
- return vma_address(vma, page_pgoff(folio, page), 1);
+ return vma_anon_address(vma, page_pgoff(folio, page), 1);
} else if (!vma->vm_file) {
return -EFAULT;
} else if (vma->vm_file->f_mapping != folio->mapping) {
@@ -1485,7 +1485,7 @@ static void __folio_set_anon(struct foli
*/
anon_vma = (void *) anon_vma + FOLIO_MAPPING_ANON;
WRITE_ONCE(folio->mapping, (struct address_space *) anon_vma);
- folio->index = linear_page_index(vma, address);
+ folio->index = linear_anon_page_index(vma, address);
}
/**
@@ -1512,8 +1512,8 @@ static void __page_check_anon_rmap(const
*/
VM_BUG_ON_FOLIO(folio_anon_vma(folio)->root != vma->anon_vma->root,
folio);
- VM_BUG_ON_PAGE(page_pgoff(folio, page) != linear_page_index(vma, address),
- page);
+ VM_BUG_ON_PAGE(page_pgoff(folio, page) !=
+ linear_anon_page_index(vma, address), page);
}
static __always_inline void __folio_add_anon_rmap(struct folio *folio,
@@ -3038,10 +3038,10 @@ static void rmap_walk_anon(struct folio
pgoff_end = pgoff_start + folio_nr_pages(folio) - 1;
anon_rmap_tree_foreach(avc, anon_vma, pgoff_start, pgoff_end) {
struct vm_area_struct *vma = avc->vma;
- unsigned long address = vma_address(vma, pgoff_start,
+ const unsigned long address = vma_anon_address(vma, pgoff_start,
folio_nr_pages(folio));
- VM_BUG_ON_VMA(address == -EFAULT, vma);
+ VM_WARN_ON_ONCE_VMA(address == -EFAULT, vma);
cond_resched();
if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg))
--- a/mm/userfaultfd.c~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/userfaultfd.c
@@ -1352,7 +1352,7 @@ static long move_present_ptes(struct mm_
}
folio_move_anon_rmap(src_folio, dst_vma);
- src_folio->index = linear_page_index(dst_vma, dst_addr);
+ src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
orig_dst_pte = folio_mk_pte(src_folio, dst_vma->vm_page_prot);
/* Set soft dirty bit so userspace can notice the pte was moved */
@@ -1428,7 +1428,7 @@ static int move_swap_pte(struct mm_struc
*/
if (src_folio) {
folio_move_anon_rmap(src_folio, dst_vma);
- src_folio->index = linear_page_index(dst_vma, dst_addr);
+ src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
} else {
/*
* Check if the swap entry is cached after acquiring the src_pte
--- a/mm/vma.c~mm-rmap-use-anon-pgoff-to-track-map_private-file-backed-anon-folios
+++ a/mm/vma.c
@@ -205,6 +205,25 @@ static void init_multi_vma_prep(struct v
}
/*
+ * Does this merge require that adjacent VMAs must have adjacent anonymous page
+ * offsets in addition to having adjacent vma->vm_pgoff?
+ *
+ * This is only required for MAP_PRIVATE-file backed mappings as the page offset
+ * for pure anonymous VMAs is equal to the anonymous page offset.
+ *
+ * Read-only shared mappings (with VMA_SHARED_BIT cleared) are always unfaulted
+ * so automatically have correct anonymous page offset (as it is always updated
+ * on remap).
+ *
+ * 'Special' mappings in the sense of VDSO, VVAR etc. have !file but would in
+ * any case not be candidates for merge nor be mergeable.
+ */
+static bool needs_adjacent_anon_pgoff(const struct vma_merge_struct *vmg)
+{
+ return vmg->file && vma_flags_is_cow_mapping(&vmg->vma_flags);
+}
+
+/*
* Return true if we can merge this (vma_flags,anon_vma,file,vm_pgoff)
* in front of (at a lower virtual address and file offset than) the vma.
*
@@ -225,6 +244,9 @@ static bool can_vma_merge_before(struct
return false;
if (vmg_end_pgoff(vmg) != vma_start_pgoff(vmg->next))
return false;
+ if (needs_adjacent_anon_pgoff(vmg) &&
+ vmg_end_anon_pgoff(vmg) != vma_start_anon_pgoff(vmg->next))
+ return false;
return true;
}
@@ -245,6 +267,9 @@ static bool can_vma_merge_after(struct v
return false;
if (vma_end_pgoff(vmg->prev) != vmg_start_pgoff(vmg))
return false;
+ if (needs_adjacent_anon_pgoff(vmg) &&
+ vma_end_anon_pgoff(vmg->prev) != vmg_start_anon_pgoff(vmg))
+ return false;
return true;
}
@@ -2048,7 +2073,12 @@ static int anon_vma_compatible(struct vm
if (!vma_flags_empty(&diff))
return false;
/* Page offset must align. */
- return vma_end_pgoff(a) == vma_start_pgoff(b);
+ if (vma_end_pgoff(a) != vma_start_pgoff(b))
+ return false;
+ /* Only reached from anon path, so either MAP_PRIVATE file or anon. */
+ if (vma_end_anon_pgoff(a) != vma_start_anon_pgoff(b))
+ return false;
+ return true;
}
/*
_
Patches currently in -mm which might be from [email protected] are
tools-testing-vma-expand-vma-merge-tests-to-assert-anon-pgoff.patch
tools-testing-selftests-mm-test-anonymous-page-offset-merge-behaviour.patch
mm-vma-only-permit-map_private-dev-zero-to-be-mapped-anonymous.patch
mm-vma-make-map_private-mapped-dev-zero-mappings-truly-anonymous.patch
tools-testing-vma-add-test-to-assert-map_private-dev-zero-is-anon.patch
tools-testing-selftests-mm-add-map_private-dev-zero-merge-tests.patch
mm-add-some-missing-includes-to-mm-local-headers.patch
maintainers-add-drivers-char-memc-to-mm-misc-memory-mapping-sections.patch