Re: [linus:master] [maple_tree] 280b792cac: will-it-scale.per_process_ops 6.0% regression
Oliver Sang <[email protected]> Sat, 23 May 2026 15:51:36 +0800
| Newsgroups | dev.linux.lists.oe-lkp,org.kernel.vger.linux-kernel,org.kvack.linux-mm |
|---|---|
| Message-ID | <ahFciGk6LfkOU2en@xsang-OptiPlex-9020> |
hi, Liam,
On Thu, May 21, 2026 at 11:45:05AM -0400, Liam R. Howlett wrote:
> On 26/05/14 03:18PM, Oliver Sang wrote:
> > hi, Liam,
> >
> > On Wed, May 13, 2026 at 08:16:42PM -0400, Liam R. Howlett wrote:
> > > On 26/05/13 03:40PM, kernel test robot wrote:
> > > >
> > > >
> > > > Hello,
> > > >
> > > > kernel test robot noticed a 6.0% regression of will-it-scale.per_process_ops on:
> > > >
> > > >
> > > > commit: 280b792cac62ddadca2935766ca870b438c86323 ("maple_tree: use maple copy node for mas_wr_split()")
> > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > >
> > > > [still regression on linus/master 5d6919055dec134de3c40167a490f33c74c12581]
> > > > [still regression on linux-next/master e98d21c170b01ddef366f023bbfcf6b31509fa83]
> > > >
> > > > testcase: will-it-scale
> > > > config: x86_64-rhel-9.4
> > > > compiler: gcc-14
> > > > test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
> > > > parameters:
> > > >
> > > > nr_task: 100%
> > > > mode: process
> > > > test: mmap2
> > > > cpufreq_governor: performance
> > > >
> > > >
> > >
> > > Thank you for the report.
> > >
> > > 48 threads on a 2 socket E5-2697 v2 looks to be 12 cores (24 threads)
> > > per cpu (so x2), or exactly one mmap2 process per hyperthread.
> >
> > this is the cpu information:
> >
> > Architecture: x86_64
> > CPU op-mode(s): 32-bit, 64-bit
> > Byte Order: Little Endian
> > Address sizes: 46 bits physical, 48 bits virtual
> > CPU(s): 48
> > On-line CPU(s) list: 0-47
> > Thread(s) per core: 2
> > Core(s) per socket: 12
> > Socket(s): 2
> > NUMA node(s): 2
> > Vendor ID: GenuineIntel
> > CPU family: 6
> > Model: 62
> > Model name: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
> > Stepping: 4
> >
> > >
> > > Is this across all process counts and peaks at 48, or just 48?
> >
> > just 48. the run script is in
> > https://download.01.org/0day-ci/archive/20260513/[email protected]/repro-script
> >
> > cd /lkp/benchmarks/will-it-scale
> > python3 ./runtest.py mmap2 295 process 0 0 48
> >
> > > Is this across many runs?
> >
> > we run 6 times for both parent and this commit, the data looks stable
> >
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json: "will-it-scale.per_process_ops": [
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json- 143595,
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json- 143474,
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json- 144104,
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json- 142796,
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json- 143081,
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json- 143623
> > 11e7f22f5e85058b09ca90e74002a3b82f50e940/matrix.json- ],
> >
> >
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json: "will-it-scale.per_process_ops": [
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json- 134451,
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json- 135089,
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json- 135080,
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json- 135039,
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json- 134082,
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json- 135301
> > 280b792cac62ddadca2935766ca870b438c86323/matrix.json- ],
> >
> >
> > >
> > > My testing didn't produce anything like this. I'll have a look into
> > > this when I can, but there isn't anything obvious that sticks out as a
> > > likely cause.
> >
> > if you want us to test any debug patch, it will be our great pleasure. thanks!
>
> It looks like the result of the shape of the tree changing. Can you try
> the attached patch against Linus' tree?
I appled your patch upon below mainline tip commit when I checked.
6779b50faa562 ("Merge tag 'pci-v7.1-fixes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci")
then build kernels with attached config.
but found a big regression introduced by your patch.
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/process/100%/debian-13-x86_64-20250902.cgz/lkp-ivb-2ep2/mmap2/will-it-scale
commit:
6779b50faa562 ("Merge tag 'pci-v7.1-fixes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci")
942596e1c1037 ("maple_tree: Restore old tree layout using new scatter-gather node copy")
6779b50faa562e6c 942596e1c1037f014638541bce4
---------------- ---------------------------
%stddev %change %stddev
\ | \
7023559 -42.9% 4007886 will-it-scale.48.processes
146323 -42.9% 83497 will-it-scale.per_process_ops
7023559 -42.9% 4007886 will-it-scale.workload
full comparison is as below [1]
however, it seems the performance regression is really recovered at commit
6779b50faa562, though the configs are not same. list the regression we reported
for refererence.
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/process/100%/debian-13-x86_64-20250902.cgz/lkp-ivb-2ep2/mmap2/will-it-scale
commit:
11e7f22f5e ("maple_tree: add cp_converged() helper")
280b792cac ("maple_tree: use maple copy node for mas_wr_split()")
11e7f22f5e85058b 280b792cac62ddadca2935766ca
---------------- ---------------------------
%stddev %change %stddev
\ | \
6885401 -6.0% 6472359 will-it-scale.48.processes
143445 -6.0% 134840 will-it-scale.per_process_ops
6885401 -6.0% 6472359 will-it-scale.workload
[1]
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/process/100%/debian-13-x86_64-20250902.cgz/lkp-ivb-2ep2/mmap2/will-it-scale
commit:
6779b50faa562 ("Merge tag 'pci-v7.1-fixes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci")
942596e1c1037 ("maple_tree: Restore old tree layout using new scatter-gather node copy")
6779b50faa562e6c 942596e1c1037f014638541bce4
---------------- ---------------------------
%stddev %change %stddev
\ | \
7023559 -42.9% 4007886 will-it-scale.48.processes
146323 -42.9% 83497 will-it-scale.per_process_ops
7023559 -42.9% 4007886 will-it-scale.workload
0.66 -12.7% 0.58 turbostat.IPC
18.34 +5.3% 19.31 turbostat.RAMWatt
0.42 +0.0 0.45 mpstat.cpu.all.irq%
2.66 +3.3 5.93 mpstat.cpu.all.soft%
14.68 -5.9 8.81 mpstat.cpu.all.usr%
14.62 -40.2% 8.75 vmstat.cpu.us
6222969 -10.4% 5575283 vmstat.memory.cache
7082 +38.9% 9835 vmstat.system.cs
2474044 ± 6% +298.0% 9845581 ± 6% numa-numastat.node0.local_node
2496926 ± 6% +295.0% 9862610 ± 6% numa-numastat.node0.numa_hit
2770972 ± 3% +255.1% 9840248 ± 5% numa-numastat.node1.local_node
2797815 ± 3% +252.9% 9872775 ± 5% numa-numastat.node1.numa_hit
224.50 ± 7% +79.4% 402.75 ± 2% perf-c2c.DRAM.local
130.88 ± 10% +104.5% 267.62 ± 5% perf-c2c.DRAM.remote
477.25 ± 8% +306.6% 1940 ± 5% perf-c2c.HITM.local
86.75 ± 17% +190.5% 252.00 ± 5% perf-c2c.HITM.remote
7083 +39.2% 9857 perf-stat.i.context-switches
243.71 -17.4% 201.31 perf-stat.i.cpu-migrations
7059 +39.2% 9824 perf-stat.ps.context-switches
242.83 -17.4% 200.61 perf-stat.ps.cpu-migrations
2432336 ± 3% -30.0% 1703846 ± 12% numa-meminfo.node1.Active
2432227 ± 3% -30.0% 1703734 ± 12% numa-meminfo.node1.Active(anon)
456558 ± 7% -60.7% 179634 ± 20% numa-meminfo.node1.Mapped
5874 ± 5% -10.6% 5254 ± 6% numa-meminfo.node1.PageTables
111108 ± 6% +9.8% 121989 ± 6% numa-meminfo.node1.SUnreclaim
2014977 ± 2% -34.3% 1324406 ± 12% numa-meminfo.node1.Shmem
0.26 ± 4% -16.5% 0.22 ± 2% perf-sched.sch_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
0.26 ± 4% -16.5% 0.22 ± 2% perf-sched.total_sch_delay.average.ms
33.78 ± 5% -41.3% 19.83 ± 2% perf-sched.total_wait_and_delay.average.ms
25847 ± 5% +73.3% 44801 ± 2% perf-sched.total_wait_and_delay.count.ms
33.51 ± 5% -41.5% 19.61 ± 2% perf-sched.total_wait_time.average.ms
33.78 ± 5% -41.3% 19.83 ± 2% perf-sched.wait_and_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
25847 ± 5% +73.3% 44801 ± 2% perf-sched.wait_and_delay.count.[unknown].[unknown].[unknown].[unknown].[unknown]
33.51 ± 5% -41.5% 19.61 ± 2% perf-sched.wait_time.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
2718564 -24.0% 2067012 meminfo.Active
2718343 -24.0% 2066792 meminfo.Active(anon)
6132342 -10.5% 5486698 meminfo.Cached
504163 ± 2% -55.1% 226137 ± 2% meminfo.Mapped
7551345 -8.6% 6901581 meminfo.Memused
11026 -5.9% 10375 meminfo.PageTables
232673 +11.1% 258411 meminfo.SUnreclaim
2036572 -31.7% 1390925 meminfo.Shmem
321293 +7.5% 345548 meminfo.Slab
7698968 -8.1% 7076107 meminfo.max_used_kB
2496631 ± 6% +295.0% 9862492 ± 6% numa-vmstat.node0.numa_hit
2473748 ± 6% +298.0% 9845463 ± 6% numa-vmstat.node0.numa_local
608040 ± 3% -30.0% 425868 ± 12% numa-vmstat.node1.nr_active_anon
113181 ± 7% -60.5% 44658 ± 20% numa-vmstat.node1.nr_mapped
1466 ± 5% -10.5% 1313 ± 6% numa-vmstat.node1.nr_page_table_pages
503740 ± 2% -34.3% 331036 ± 12% numa-vmstat.node1.nr_shmem
27840 ± 6% +9.9% 30585 ± 6% numa-vmstat.node1.nr_slab_unreclaimable
608040 ± 3% -30.0% 425868 ± 12% numa-vmstat.node1.nr_zone_active_anon
2797544 ± 3% +252.9% 9872464 ± 5% numa-vmstat.node1.numa_hit
2770701 ± 3% +255.1% 9839937 ± 5% numa-vmstat.node1.numa_local
98407 ± 8% -16.6% 82034 ± 12% sched_debug.cfs_rq:/.avg_vruntime.stddev
584720 ± 55% +82.9% 1069238 ± 20% sched_debug.cfs_rq:/.left_deadline.stddev
584716 ± 55% +82.9% 1069230 ± 20% sched_debug.cfs_rq:/.left_vruntime.stddev
584717 ± 55% +82.9% 1069231 ± 20% sched_debug.cfs_rq:/.right_vruntime.stddev
98406 ± 8% -16.6% 82034 ± 12% sched_debug.cfs_rq:/.zero_vruntime.stddev
3696 ± 22% -53.0% 1736 ± 28% sched_debug.cpu.curr->pid.min
1080 ± 8% +22.1% 1320 ± 7% sched_debug.cpu.curr->pid.stddev
24404 +36.7% 33355 sched_debug.cpu.nr_switches.avg
35594 ± 5% +26.6% 45073 ± 6% sched_debug.cpu.nr_switches.max
19226 +47.9% 28429 sched_debug.cpu.nr_switches.min
679722 -24.0% 516709 proc-vmstat.nr_active_anon
1446664 +1.1% 1462897 proc-vmstat.nr_dirty_background_threshold
2896867 +1.1% 2929372 proc-vmstat.nr_dirty_threshold
1533216 -10.5% 1371700 proc-vmstat.nr_file_pages
14575614 +1.1% 14738179 proc-vmstat.nr_free_pages
126756 ± 2% -55.4% 56505 ± 2% proc-vmstat.nr_mapped
2757 -6.0% 2592 proc-vmstat.nr_page_table_pages
509273 -31.7% 347755 proc-vmstat.nr_shmem
22155 -1.7% 21783 proc-vmstat.nr_slab_reclaimable
58291 +10.7% 64524 proc-vmstat.nr_slab_unreclaimable
679722 -24.0% 516709 proc-vmstat.nr_zone_active_anon
5296571 +272.6% 19736811 proc-vmstat.numa_hit
5246846 +275.2% 19687252 proc-vmstat.numa_local
9465911 +307.2% 38547209 proc-vmstat.pgalloc_normal
8803386 ± 2% +332.6% 38087100 proc-vmstat.pgfree
19.27 -5.7 13.54 perf-profile.calltrace.cycles-pp.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
15.16 -5.1 10.07 perf-profile.calltrace.cycles-pp.unmap_region.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
54.50 -4.2 50.33 perf-profile.calltrace.cycles-pp.__mmap
8.54 -2.9 5.66 perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
8.04 -2.8 5.22 perf-profile.calltrace.cycles-pp.__zap_vma_range.unmap_vmas.unmap_region.vms_complete_munmap_vmas.do_vmi_align_munmap
9.39 -2.6 6.83 perf-profile.calltrace.cycles-pp.__mmap_complete.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
6.62 -2.5 4.12 perf-profile.calltrace.cycles-pp.zap_pmd_range.__zap_vma_range.unmap_vmas.unmap_region.vms_complete_munmap_vmas
8.69 -2.3 6.34 perf-profile.calltrace.cycles-pp.perf_event_mmap.__mmap_complete.__mmap_region.do_mmap.vm_mmap_pgoff
8.36 -2.3 6.10 perf-profile.calltrace.cycles-pp.perf_event_mmap_event.perf_event_mmap.__mmap_complete.__mmap_region.do_mmap
5.66 -1.9 3.72 perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
3.90 -1.7 2.20 perf-profile.calltrace.cycles-pp.entry_SYSRETQ_unsafe_stack.__mmap
3.76 -1.6 2.13 perf-profile.calltrace.cycles-pp.entry_SYSRETQ_unsafe_stack.__munmap
2.99 ± 3% -1.6 1.39 perf-profile.calltrace.cycles-pp.mas_preallocate.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
3.68 -1.3 2.39 perf-profile.calltrace.cycles-pp.free_pgd_range.free_pgtables.unmap_region.vms_complete_munmap_vmas.do_vmi_align_munmap
3.38 -1.2 2.18 perf-profile.calltrace.cycles-pp.free_p4d_range.free_pgd_range.free_pgtables.unmap_region.vms_complete_munmap_vmas
2.86 -1.0 1.81 perf-profile.calltrace.cycles-pp.free_pud_range.free_p4d_range.free_pgd_range.free_pgtables.unmap_region
6.25 -1.0 5.24 perf-profile.calltrace.cycles-pp.__get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
5.95 -1.0 4.95 perf-profile.calltrace.cycles-pp.shmem_get_unmapped_area.__get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
2.77 ± 2% -0.9 1.85 ± 2% perf-profile.calltrace.cycles-pp.d_path.perf_event_mmap_event.perf_event_mmap.__mmap_complete.__mmap_region
2.41 -0.8 1.65 perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.__zap_vma_range.unmap_vmas.unmap_region
1.53 -0.7 0.85 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__munmap
1.50 -0.6 0.85 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__mmap
4.86 -0.6 4.23 perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.__get_unmapped_area.do_mmap.vm_mmap_pgoff
1.67 ± 4% -0.6 1.05 ± 4% perf-profile.calltrace.cycles-pp.prepend_path.d_path.perf_event_mmap_event.perf_event_mmap.__mmap_complete
1.37 -0.6 0.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__munmap
1.36 -0.6 0.76 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__mmap
1.78 -0.6 1.22 perf-profile.calltrace.cycles-pp.perf_iterate_sb.perf_event_mmap_event.perf_event_mmap.__mmap_complete.__mmap_region
1.69 -0.6 1.13 perf-profile.calltrace.cycles-pp.shmem_mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
4.11 -0.5 3.60 perf-profile.calltrace.cycles-pp.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.__get_unmapped_area.do_mmap
1.23 -0.5 0.72 perf-profile.calltrace.cycles-pp.mas_walk.mas_find.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
1.38 -0.5 0.89 perf-profile.calltrace.cycles-pp.touch_atime.shmem_mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff
1.37 -0.5 0.89 perf-profile.calltrace.cycles-pp.mas_find.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
3.96 -0.5 3.49 perf-profile.calltrace.cycles-pp.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.__get_unmapped_area
1.46 -0.5 1.01 perf-profile.calltrace.cycles-pp.mas_find.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
1.10 -0.4 0.67 perf-profile.calltrace.cycles-pp.security_vm_enough_memory_mm.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
1.11 -0.4 0.73 perf-profile.calltrace.cycles-pp.unlink_file_vma_batch_process.free_pgtables.unmap_region.vms_complete_munmap_vmas.do_vmi_align_munmap
0.97 -0.4 0.60 perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.shmem_mmap_prepare.__mmap_region.do_mmap
0.95 -0.4 0.58 perf-profile.calltrace.cycles-pp.vma_merge_new_range.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
2.41 -0.3 2.07 perf-profile.calltrace.cycles-pp.vm_area_alloc.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
1.71 -0.3 1.40 perf-profile.calltrace.cycles-pp.__build_id_parse.perf_event_mmap_event.perf_event_mmap.__mmap_complete.__mmap_region
0.80 -0.3 0.55 perf-profile.calltrace.cycles-pp.pte_offset_map_lock.zap_pte_range.zap_pmd_range.__zap_vma_range.unmap_vmas
1.20 -0.2 1.00 perf-profile.calltrace.cycles-pp.mas_store_gfp.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
0.81 -0.2 0.63 perf-profile.calltrace.cycles-pp.mas_wr_store_type.mas_preallocate.__mmap_new_vma.__mmap_region.do_mmap
1.97 -0.2 1.79 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region.do_mmap
0.76 -0.2 0.60 perf-profile.calltrace.cycles-pp.mas_walk.mas_find.__mmap_region.do_mmap.vm_mmap_pgoff
1.15 -0.1 1.01 perf-profile.calltrace.cycles-pp.freader_fetch.__build_id_parse.perf_event_mmap_event.perf_event_mmap.__mmap_complete
1.48 -0.1 1.36 perf-profile.calltrace.cycles-pp.mas_rev_awalk.mas_empty_area_rev.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown
0.90 -0.1 0.78 perf-profile.calltrace.cycles-pp.freader_get_folio.freader_fetch.__build_id_parse.perf_event_mmap_event.perf_event_mmap
0.77 -0.1 0.66 perf-profile.calltrace.cycles-pp.khugepaged_enter_vma.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
1.93 ± 2% -0.1 1.82 perf-profile.calltrace.cycles-pp.kmem_cache_free.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
2.20 -0.1 2.11 perf-profile.calltrace.cycles-pp.mas_empty_area_rev.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area
0.66 -0.1 0.58 ± 2% perf-profile.calltrace.cycles-pp.__filemap_get_folio_mpol.freader_get_folio.freader_fetch.__build_id_parse.perf_event_mmap_event
1.17 ± 3% -0.1 1.10 perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.cmd_record
1.17 ± 3% -0.1 1.10 perf-profile.calltrace.cycles-pp.cmd_record
1.17 ± 3% -0.1 1.10 perf-profile.calltrace.cycles-pp.record__finish_output.cmd_record
5.13 +0.1 5.20 perf-profile.calltrace.cycles-pp.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
44.84 +0.2 45.05 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
0.70 +0.4 1.07 perf-profile.calltrace.cycles-pp.mas_prev_slot.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
0.20 ±129% +0.4 0.60 ± 2% perf-profile.calltrace.cycles-pp.ordered_events__deliver_event.__ordered_events__flush.perf_session__process_user_event.perf_session__process_events.record__finish_output
0.22 ±129% +0.4 0.64 ± 2% perf-profile.calltrace.cycles-pp.__ordered_events__flush.perf_session__process_user_event.perf_session__process_events.record__finish_output.cmd_record
0.22 ±129% +0.4 0.64 ± 2% perf-profile.calltrace.cycles-pp.perf_session__process_user_event.perf_session__process_events.record__finish_output.cmd_record
0.13 ±173% +0.5 0.59 ± 2% perf-profile.calltrace.cycles-pp.perf_session__deliver_event.ordered_events__deliver_event.__ordered_events__flush.perf_session__process_user_event.perf_session__process_events
43.80 +0.5 44.30 perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
0.00 +0.5 0.52 ± 2% perf-profile.calltrace.cycles-pp.node_finalise.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.memcpy_orig.node_copy.cp_data_write.mas_wr_spanning_store.mas_store_gfp
0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.memcpy_orig.node_copy.cp_data_write.mas_wr_split.mas_store_prealloc
0.89 +0.6 1.47 perf-profile.calltrace.cycles-pp.mas_find.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
0.00 +0.7 0.66 perf-profile.calltrace.cycles-pp.mas_next_node.mas_next_slot.mas_find.vms_gather_munmap_vmas.do_vmi_align_munmap
0.65 +0.7 1.31 perf-profile.calltrace.cycles-pp.mas_next_slot.mas_find.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
0.00 +0.7 0.67 ± 2% perf-profile.calltrace.cycles-pp.__kfree_rcu_sheaf.kvfree_call_rcu.mas_topiary_replace.mas_wr_spanning_store.mas_store_gfp
0.00 +0.8 0.78 perf-profile.calltrace.cycles-pp.node_finalise.mas_wr_split.mas_store_prealloc.__mmap_new_vma.__mmap_region
0.00 +0.8 0.82 ± 2% perf-profile.calltrace.cycles-pp.__slab_free.__kmem_cache_free_bulk.rcu_free_sheaf.rcu_do_batch.rcu_core
42.40 +1.0 43.37 perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
0.00 +1.0 1.00 ± 2% perf-profile.calltrace.cycles-pp.kvfree_call_rcu.mas_topiary_replace.mas_wr_split.mas_store_prealloc.__mmap_new_vma
0.00 +1.3 1.25 perf-profile.calltrace.cycles-pp.kvfree_call_rcu.mas_topiary_replace.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap
41.16 +1.3 42.42 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +1.3 1.26 ± 2% perf-profile.calltrace.cycles-pp.__kmem_cache_free_bulk.rcu_free_sheaf.rcu_do_batch.rcu_core.handle_softirqs
0.00 +1.3 1.31 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__refill_objects_node.refill_objects.__prefill_sheaf_pfmemalloc
0.00 +1.4 1.40 perf-profile.calltrace.cycles-pp.rcu_free_sheaf.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd
0.00 +1.4 1.44 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__refill_objects_node.refill_objects.__prefill_sheaf_pfmemalloc.kmem_cache_prefill_sheaf
0.00 +1.5 1.45 ± 2% perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn
0.00 +1.5 1.46 ± 2% perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread
0.00 +1.5 1.47 perf-profile.calltrace.cycles-pp.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
0.00 +1.5 1.47 perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
45.00 +1.5 46.50 perf-profile.calltrace.cycles-pp.__munmap
0.00 +1.5 1.50 ± 2% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
0.00 +1.7 1.73 perf-profile.calltrace.cycles-pp.node_copy.cp_data_write.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap
0.00 +1.7 1.73 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
0.00 +1.7 1.73 perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
0.00 +1.7 1.73 perf-profile.calltrace.cycles-pp.ret_from_fork_asm
0.00 +1.8 1.82 perf-profile.calltrace.cycles-pp.node_copy.cp_data_write.mas_wr_split.mas_store_prealloc.__mmap_new_vma
0.00 +2.0 1.95 perf-profile.calltrace.cycles-pp.dst_setup.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
33.18 +2.6 35.81 perf-profile.calltrace.cycles-pp.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
0.00 +2.9 2.88 perf-profile.calltrace.cycles-pp.cp_data_write.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
0.00 +3.0 3.03 perf-profile.calltrace.cycles-pp.dst_setup.mas_wr_split.mas_store_prealloc.__mmap_new_vma.__mmap_region
0.00 +3.3 3.31 perf-profile.calltrace.cycles-pp.mas_topiary_replace.mas_wr_split.mas_store_prealloc.__mmap_new_vma.__mmap_region
0.00 +3.6 3.59 perf-profile.calltrace.cycles-pp.mas_topiary_replace.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
0.00 +3.9 3.87 perf-profile.calltrace.cycles-pp.__refill_objects_node.refill_objects.__prefill_sheaf_pfmemalloc.kmem_cache_prefill_sheaf.mas_store_gfp
0.00 +3.9 3.91 perf-profile.calltrace.cycles-pp.cp_data_write.mas_wr_split.mas_store_prealloc.__mmap_new_vma.__mmap_region
0.00 +4.6 4.64 perf-profile.calltrace.cycles-pp.refill_objects.__prefill_sheaf_pfmemalloc.kmem_cache_prefill_sheaf.mas_store_gfp.do_vmi_align_munmap
0.00 +4.7 4.71 perf-profile.calltrace.cycles-pp.__prefill_sheaf_pfmemalloc.kmem_cache_prefill_sheaf.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
0.00 +5.1 5.10 perf-profile.calltrace.cycles-pp.kmem_cache_prefill_sheaf.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
36.19 +5.6 41.79 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
36.00 +5.7 41.69 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
35.15 +6.0 41.13 perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
35.01 +6.0 41.03 perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
34.08 +6.3 40.39 perf-profile.calltrace.cycles-pp.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
32.43 +6.8 39.26 perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
12.68 +8.9 21.54 perf-profile.calltrace.cycles-pp.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
4.77 +11.2 16.00 perf-profile.calltrace.cycles-pp.mas_store_prealloc.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
7.25 +12.6 19.89 perf-profile.calltrace.cycles-pp.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
0.00 +13.7 13.69 perf-profile.calltrace.cycles-pp.mas_wr_spanning_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
0.00 +14.8 14.81 perf-profile.calltrace.cycles-pp.mas_wr_split.mas_store_prealloc.__mmap_new_vma.__mmap_region.do_mmap
8.17 -8.2 0.00 perf-profile.children.cycles-pp.mas_wr_node_store
19.48 -5.8 13.71 perf-profile.children.cycles-pp.vms_complete_munmap_vmas
15.18 -5.1 10.08 perf-profile.children.cycles-pp.unmap_region
9.02 -4.0 4.98 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
54.00 -3.6 50.44 perf-profile.children.cycles-pp.__mmap
5.18 -3.4 1.80 perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
8.56 -2.9 5.68 perf-profile.children.cycles-pp.unmap_vmas
8.07 -2.8 5.24 perf-profile.children.cycles-pp.__zap_vma_range
9.41 -2.6 6.84 perf-profile.children.cycles-pp.__mmap_complete
6.64 -2.5 4.13 perf-profile.children.cycles-pp.zap_pmd_range
8.70 -2.3 6.36 perf-profile.children.cycles-pp.perf_event_mmap
8.41 -2.3 6.13 perf-profile.children.cycles-pp.perf_event_mmap_event
5.75 -2.0 3.78 perf-profile.children.cycles-pp.free_pgtables
3.02 ± 3% -1.6 1.41 perf-profile.children.cycles-pp.mas_preallocate
3.48 -1.5 1.94 perf-profile.children.cycles-pp.entry_SYSCALL_64
3.09 -1.4 1.74 perf-profile.children.cycles-pp.syscall_return_via_sysret
3.70 -1.3 2.40 perf-profile.children.cycles-pp.free_pgd_range
3.39 -1.2 2.19 perf-profile.children.cycles-pp.free_p4d_range
2.86 -1.0 1.82 perf-profile.children.cycles-pp.free_pud_range
6.26 -1.0 5.24 perf-profile.children.cycles-pp.__get_unmapped_area
5.96 -1.0 4.95 perf-profile.children.cycles-pp.shmem_get_unmapped_area
2.78 ± 2% -0.9 1.85 ± 2% perf-profile.children.cycles-pp.d_path
1.85 -0.8 1.07 perf-profile.children.cycles-pp.mas_wr_store_type
2.43 -0.8 1.66 perf-profile.children.cycles-pp.zap_pte_range
2.16 -0.7 1.47 perf-profile.children.cycles-pp.mas_walk
4.91 -0.7 4.26 perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
1.68 ± 4% -0.6 1.05 ± 4% perf-profile.children.cycles-pp.prepend_path
4.81 -0.6 4.24 perf-profile.children.cycles-pp.mas_find
1.80 -0.6 1.24 perf-profile.children.cycles-pp.perf_iterate_sb
1.72 -0.6 1.16 perf-profile.children.cycles-pp.shmem_mmap_prepare
0.65 -0.5 0.13 ± 3% perf-profile.children.cycles-pp.mas_update_gap
4.12 -0.5 3.61 perf-profile.children.cycles-pp.vm_unmapped_area
1.13 -0.5 0.63 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
1.39 -0.5 0.90 perf-profile.children.cycles-pp.touch_atime
3.98 -0.5 3.50 perf-profile.children.cycles-pp.unmapped_area_topdown
1.16 -0.5 0.70 perf-profile.children.cycles-pp.security_vm_enough_memory_mm
1.00 -0.4 0.62 ± 2% perf-profile.children.cycles-pp.atime_needs_update
0.99 -0.4 0.60 perf-profile.children.cycles-pp.vma_merge_new_range
1.12 -0.4 0.74 perf-profile.children.cycles-pp.unlink_file_vma_batch_process
1.05 -0.4 0.68 perf-profile.children.cycles-pp.__vma_start_write
2.42 -0.3 2.08 perf-profile.children.cycles-pp.vm_area_alloc
1.73 -0.3 1.41 perf-profile.children.cycles-pp.__build_id_parse
0.89 ± 3% -0.3 0.59 ± 3% perf-profile.children.cycles-pp.kfree
0.92 ± 2% -0.3 0.63 ± 2% perf-profile.children.cycles-pp.prepend_copy
1.51 -0.3 1.24 perf-profile.children.cycles-pp.memcpy_orig
0.68 -0.3 0.41 perf-profile.children.cycles-pp.__vma_start_exclude_readers
0.59 ± 2% -0.3 0.33 ± 10% perf-profile.children.cycles-pp.current_time
0.82 -0.3 0.56 perf-profile.children.cycles-pp.pte_offset_map_lock
0.83 ± 2% -0.3 0.57 ± 2% perf-profile.children.cycles-pp.copy_from_kernel_nofault
0.54 ± 8% -0.2 0.33 ± 3% perf-profile.children.cycles-pp.perf_event_mmap_output
0.61 ± 3% -0.2 0.40 ± 3% perf-profile.children.cycles-pp.fget
0.58 ± 4% -0.2 0.37 ± 4% perf-profile.children.cycles-pp.ordered_events__queue
0.58 ± 4% -0.2 0.36 ± 5% perf-profile.children.cycles-pp.queue_event
0.59 ± 2% -0.2 0.38 ± 3% perf-profile.children.cycles-pp.vma_set_page_prot
0.64 ± 3% -0.2 0.44 ± 3% perf-profile.children.cycles-pp.process_simple
0.50 ± 3% -0.2 0.31 ± 3% perf-profile.children.cycles-pp.x64_sys_call
0.61 -0.2 0.43 perf-profile.children.cycles-pp.up_write
0.61 -0.2 0.45 perf-profile.children.cycles-pp.__thp_vma_allowable_orders
0.41 ± 2% -0.2 0.25 perf-profile.children.cycles-pp.fput
0.41 ± 2% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.down_write
0.40 ± 3% -0.1 0.25 ± 2% perf-profile.children.cycles-pp.up_read
1.16 -0.1 1.02 perf-profile.children.cycles-pp.freader_fetch
0.38 -0.1 0.25 perf-profile.children.cycles-pp.can_vma_merge_right
0.30 ± 2% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.static_key_count
1.48 -0.1 1.36 perf-profile.children.cycles-pp.mas_rev_awalk
0.38 -0.1 0.26 ± 2% perf-profile.children.cycles-pp.remove_vma
0.24 ± 2% -0.1 0.12 ± 3% perf-profile.children.cycles-pp.unlink_anon_vmas
0.80 -0.1 0.69 perf-profile.children.cycles-pp.khugepaged_enter_vma
0.29 -0.1 0.18 ± 2% perf-profile.children.cycles-pp.cap_vm_enough_memory
0.91 -0.1 0.79 perf-profile.children.cycles-pp.freader_get_folio
0.30 ± 2% -0.1 0.18 ± 3% perf-profile.children.cycles-pp.mmap_region
0.35 ± 4% -0.1 0.24 ± 2% perf-profile.children.cycles-pp.strlen
0.30 ± 2% -0.1 0.19 perf-profile.children.cycles-pp.downgrade_write
0.45 ± 5% -0.1 0.34 ± 4% perf-profile.children.cycles-pp.down_write_killable
0.26 -0.1 0.15 ± 2% perf-profile.children.cycles-pp.freader_init_from_file
1.95 ± 2% -0.1 1.84 perf-profile.children.cycles-pp.kmem_cache_free
0.36 -0.1 0.25 ± 2% perf-profile.children.cycles-pp.may_expand_vm
0.33 ± 3% -0.1 0.22 ± 4% perf-profile.children.cycles-pp.vma_interval_tree_remove
0.34 ± 2% -0.1 0.24 ± 3% perf-profile.children.cycles-pp._raw_spin_lock
0.23 ± 3% -0.1 0.13 ± 6% perf-profile.children.cycles-pp.prandom_u32_state
0.32 -0.1 0.22 ± 2% perf-profile.children.cycles-pp.tlb_gather_mmu
0.29 ± 4% -0.1 0.19 ± 5% perf-profile.children.cycles-pp.uprobe_mmap
0.37 ± 3% -0.1 0.27 ± 6% perf-profile.children.cycles-pp.vm_get_page_prot
2.24 -0.1 2.14 perf-profile.children.cycles-pp.mas_empty_area_rev
0.26 -0.1 0.16 ± 2% perf-profile.children.cycles-pp.__vm_enough_memory
0.36 -0.1 0.26 perf-profile.children.cycles-pp.mmap_action_complete
0.24 ± 4% -0.1 0.16 ± 13% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg
0.32 -0.1 0.24 ± 2% perf-profile.children.cycles-pp.mas_next_range
0.68 -0.1 0.60 ± 2% perf-profile.children.cycles-pp.__filemap_get_folio_mpol
1.17 ± 3% -0.1 1.10 perf-profile.children.cycles-pp.perf_session__process_events
1.17 ± 3% -0.1 1.10 perf-profile.children.cycles-pp.record__finish_output
0.19 ± 4% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.cap_capable
0.30 ± 2% -0.1 0.23 perf-profile.children.cycles-pp.tlb_finish_mmu
1.21 ± 3% -0.1 1.15 perf-profile.children.cycles-pp.cmd_record
0.19 ± 2% -0.1 0.13 ± 4% perf-profile.children.cycles-pp.__x64_sys_mmap
0.20 ± 2% -0.1 0.13 ± 3% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.34 ± 2% -0.1 0.27 perf-profile.children.cycles-pp.userfaultfd_unmap_complete
0.14 ± 3% -0.1 0.08 perf-profile.children.cycles-pp.unlink_file_vma_batch_add
0.10 -0.1 0.04 ± 57% perf-profile.children.cycles-pp.can_vma_merge_left
0.19 ± 3% -0.1 0.13 ± 3% perf-profile.children.cycles-pp.vma_wants_writenotify
0.17 -0.1 0.11 perf-profile.children.cycles-pp.ksm_vma_flags
0.22 ± 2% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.vm_area_free
0.16 ± 2% -0.1 0.11 ± 3% perf-profile.children.cycles-pp.__pte_offset_map
0.23 ± 2% -0.1 0.18 ± 3% perf-profile.children.cycles-pp.testcase
0.20 ± 4% -0.1 0.15 ± 2% perf-profile.children.cycles-pp.uprobe_munmap
0.15 ± 4% -0.1 0.10 ± 3% perf-profile.children.cycles-pp.vma_is_shared_writable
0.12 ± 5% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.__rb_insert_augmented
0.23 ± 3% -0.0 0.18 ± 3% perf-profile.children.cycles-pp.ima_file_mmap
0.18 ± 16% -0.0 0.13 ± 16% perf-profile.children.cycles-pp.memfd_check_seals_mmap
0.15 ± 2% -0.0 0.10 ± 6% perf-profile.children.cycles-pp.mm_get_unmapped_area
0.13 ± 4% -0.0 0.08 ± 7% perf-profile.children.cycles-pp.rb_next
0.09 ± 6% -0.0 0.04 ± 37% perf-profile.children.cycles-pp.flush_tlb_batched_pending
0.17 ± 3% -0.0 0.13 ± 3% perf-profile.children.cycles-pp.mas_prev_range
0.48 ± 2% -0.0 0.43 perf-profile.children.cycles-pp.mas_prev
0.10 ± 3% -0.0 0.06 perf-profile.children.cycles-pp._raw_spin_unlock
0.20 ± 5% -0.0 0.16 ± 3% perf-profile.children.cycles-pp.vma_is_special_huge
0.12 ± 4% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.mas_next_setup
0.29 ± 2% -0.0 0.25 ± 2% perf-profile.children.cycles-pp.mas_wr_store_entry
0.07 ± 7% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.finish_rcuwait
0.07 ± 5% -0.0 0.04 ± 37% perf-profile.children.cycles-pp.__mt_destroy
0.22 ± 2% -0.0 0.19 ± 2% perf-profile.children.cycles-pp.copy_from_kernel_nofault_allowed
0.09 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.prepend
0.24 ± 2% -0.0 0.22 perf-profile.children.cycles-pp.mas_prev_setup
0.09 -0.0 0.08 ± 6% perf-profile.children.cycles-pp.get_align_mask
0.10 ± 3% -0.0 0.09 perf-profile.children.cycles-pp.mmap_action_prepare
0.32 +0.0 0.34 perf-profile.children.cycles-pp.tick_nohz_handler
0.29 +0.0 0.30 perf-profile.children.cycles-pp.update_process_times
0.07 ± 4% +0.0 0.09 ± 8% perf-profile.children.cycles-pp.trylock_stock
0.06 ± 11% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.maps__fixup_overlap_and_insert
0.05 ± 39% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.machine__findnew_thread
0.10 +0.0 0.12 ± 2% perf-profile.children.cycles-pp.__kmalloc_noprof
0.11 +0.0 0.13 ± 2% perf-profile.children.cycles-pp.__alloc_empty_sheaf
0.05 ± 6% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.mmap@plt
0.06 ± 8% +0.0 0.08 ± 4% perf-profile.children.cycles-pp.rcu_cblist_dequeue
0.13 ± 10% +0.0 0.16 ± 5% perf-profile.children.cycles-pp.map__new
0.05 +0.0 0.09 perf-profile.children.cycles-pp.schedule
0.14 ± 3% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.is_vmalloc_addr
0.08 ± 6% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.__call_rcu_common
0.18 ± 12% +0.0 0.22 ± 11% perf-profile.children.cycles-pp.__refill_obj_stock
0.05 ± 6% +0.0 0.10 perf-profile.children.cycles-pp.__schedule
5.19 +0.0 5.24 perf-profile.children.cycles-pp.vms_gather_munmap_vmas
0.00 +0.1 0.06 ± 5% perf-profile.children.cycles-pp.add_callchain_ip
0.28 ± 9% +0.1 0.34 ± 3% perf-profile.children.cycles-pp.machine__process_mmap2_event
0.05 ± 42% +0.1 0.12 ± 11% perf-profile.children.cycles-pp.build_id__mark_dso_hit
0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.get_page_from_freelist
0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.barn_put_empty_sheaf
0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.thread__resolve_callchain_sample
0.04 ± 79% +0.1 0.11 ± 12% perf-profile.children.cycles-pp.sample__for_each_callchain_node
0.00 +0.1 0.07 perf-profile.children.cycles-pp.build_detached_freelist
0.00 +0.1 0.07 ± 4% perf-profile.children.cycles-pp.__alloc_frozen_pages_noprof
0.03 ±101% +0.1 0.10 ± 12% perf-profile.children.cycles-pp.__thread__resolve_callchain
1.53 +0.1 1.60 perf-profile.children.cycles-pp.mas_prev_slot
0.15 ± 3% +0.1 0.22 ± 6% perf-profile.children.cycles-pp.xas_load
0.07 ± 5% +0.1 0.17 ± 2% perf-profile.children.cycles-pp.barn_get_empty_sheaf
0.10 +0.1 0.20 perf-profile.children.cycles-pp.__pi_memcpy
0.07 ± 6% +0.1 0.17 ± 2% perf-profile.children.cycles-pp.__refill_objects_any
0.00 +0.1 0.11 perf-profile.children.cycles-pp.alloc_from_new_slab
0.51 ± 10% +0.1 0.64 ± 2% perf-profile.children.cycles-pp.__ordered_events__flush
0.51 ± 10% +0.1 0.64 ± 2% perf-profile.children.cycles-pp.perf_session__process_user_event
0.46 ± 9% +0.1 0.60 ± 2% perf-profile.children.cycles-pp.perf_session__deliver_event
0.47 ± 10% +0.1 0.60 ± 2% perf-profile.children.cycles-pp.ordered_events__deliver_event
0.25 ± 4% +0.1 0.39 perf-profile.children.cycles-pp.__account_obj_stock
0.06 +0.2 0.21 ± 2% perf-profile.children.cycles-pp.kmem_cache_free_bulk
0.06 +0.2 0.21 ± 2% perf-profile.children.cycles-pp.kvfree_rcu_bulk
0.06 +0.2 0.21 ± 2% perf-profile.children.cycles-pp.kfree_rcu_work
0.06 ± 7% +0.2 0.22 ± 2% perf-profile.children.cycles-pp.process_one_work
0.32 ± 2% +0.2 0.48 perf-profile.children.cycles-pp.__rcu_free_sheaf_prepare
0.07 ± 7% +0.2 0.22 ± 2% perf-profile.children.cycles-pp.worker_thread
0.04 ± 38% +0.2 0.24 ± 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.09 ± 5% +0.2 0.31 ± 3% perf-profile.children.cycles-pp.setup_object
0.66 ± 3% +0.2 0.89 perf-profile.children.cycles-pp.mas_wr_walk_descend
0.10 ± 4% +0.3 0.38 ± 2% perf-profile.children.cycles-pp.shuffle_freelist
1.70 +0.3 1.99 perf-profile.children.cycles-pp.mas_next_slot
0.12 ± 5% +0.3 0.46 ± 2% perf-profile.children.cycles-pp.allocate_slab
0.76 ± 3% +0.4 1.14 ± 2% perf-profile.children.cycles-pp.__kfree_rcu_sheaf
0.00 +0.4 0.39 perf-profile.children.cycles-pp.kmem_cache_return_sheaf
43.84 +0.5 44.31 perf-profile.children.cycles-pp.ksys_mmap_pgoff
1.87 +0.5 2.35 perf-profile.children.cycles-pp.kvfree_call_rcu
0.00 +0.5 0.51 perf-profile.children.cycles-pp.mas_prev_node
0.00 +0.5 0.52 perf-profile.children.cycles-pp.mas_wr_walk_index
0.00 +0.7 0.66 perf-profile.children.cycles-pp.kmem_cache_alloc_from_sheaf_noprof
0.00 +0.7 0.67 perf-profile.children.cycles-pp.mas_next_node
0.00 +0.7 0.67 perf-profile.children.cycles-pp.mas_ascend
0.00 +0.7 0.71 perf-profile.children.cycles-pp.multi_src_setup
42.42 +1.0 43.39 perf-profile.children.cycles-pp.vm_mmap_pgoff
41.20 +1.2 42.45 perf-profile.children.cycles-pp.do_mmap
0.22 ± 3% +1.3 1.47 perf-profile.children.cycles-pp.run_ksoftirqd
0.23 ± 2% +1.3 1.50 ± 2% perf-profile.children.cycles-pp.smpboot_thread_fn
0.31 ± 2% +1.4 1.73 perf-profile.children.cycles-pp.ret_from_fork
0.31 ± 2% +1.4 1.73 perf-profile.children.cycles-pp.ret_from_fork_asm
0.30 ± 2% +1.4 1.73 perf-profile.children.cycles-pp.kthread
0.00 +1.6 1.64 perf-profile.children.cycles-pp.node_finalise
44.66 +2.0 46.68 perf-profile.children.cycles-pp.__munmap
1.47 +2.6 4.04 perf-profile.children.cycles-pp.__refill_objects_node
33.33 +2.6 35.93 perf-profile.children.cycles-pp.__mmap_region
2.63 +2.7 5.35 perf-profile.children.cycles-pp.__irq_exit_rcu
1.76 +2.7 4.48 perf-profile.children.cycles-pp.__slab_free
3.08 +2.7 5.81 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
3.13 +2.7 5.87 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.50 ± 2% +2.9 3.41 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.64 +3.0 4.67 perf-profile.children.cycles-pp.refill_objects
0.81 ± 2% +3.2 3.98 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
2.24 +3.6 5.89 perf-profile.children.cycles-pp.__kmem_cache_free_bulk
0.00 +3.7 3.67 perf-profile.children.cycles-pp.node_copy
2.60 +3.9 6.46 perf-profile.children.cycles-pp.rcu_free_sheaf
2.75 +3.9 6.69 perf-profile.children.cycles-pp.rcu_do_batch
2.78 +4.0 6.73 perf-profile.children.cycles-pp.rcu_core
2.82 +4.0 6.79 perf-profile.children.cycles-pp.handle_softirqs
0.00 +4.7 4.71 perf-profile.children.cycles-pp.__prefill_sheaf_pfmemalloc
0.00 +5.1 5.09 perf-profile.children.cycles-pp.dst_setup
0.00 +5.5 5.47 perf-profile.children.cycles-pp.kmem_cache_prefill_sheaf
81.39 +5.7 87.09 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
80.99 +5.9 86.88 perf-profile.children.cycles-pp.do_syscall_64
35.17 +6.0 41.14 perf-profile.children.cycles-pp.__x64_sys_munmap
35.04 +6.0 41.05 perf-profile.children.cycles-pp.__vm_munmap
34.11 +6.3 40.41 perf-profile.children.cycles-pp.do_vmi_munmap
32.45 +6.8 39.28 perf-profile.children.cycles-pp.do_vmi_align_munmap
0.00 +6.8 6.84 perf-profile.children.cycles-pp.cp_data_write
0.00 +6.9 6.94 perf-profile.children.cycles-pp.mas_topiary_replace
12.75 +8.8 21.59 perf-profile.children.cycles-pp.__mmap_new_vma
4.79 +11.2 16.02 perf-profile.children.cycles-pp.mas_store_prealloc
8.52 +12.4 20.93 perf-profile.children.cycles-pp.mas_store_gfp
0.00 +13.8 13.76 perf-profile.children.cycles-pp.mas_wr_spanning_store
0.00 +14.9 14.87 perf-profile.children.cycles-pp.mas_wr_split
8.95 -4.0 4.94 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
4.07 -1.8 2.32 perf-profile.self.cycles-pp.zap_pmd_range
1.87 ± 3% -1.4 0.48 ± 3% perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
3.09 -1.4 1.72 perf-profile.self.cycles-pp.entry_SYSCALL_64
3.04 -1.3 1.72 perf-profile.self.cycles-pp.syscall_return_via_sysret
2.76 -1.1 1.70 perf-profile.self.cycles-pp.free_pud_range
2.75 -0.8 1.92 perf-profile.self.cycles-pp.__mmap_region
2.03 -0.7 1.35 perf-profile.self.cycles-pp.mas_walk
1.14 -0.6 0.50 ± 2% perf-profile.self.cycles-pp.mas_wr_store_type
0.63 -0.5 0.12 ± 4% perf-profile.self.cycles-pp.mas_update_gap
1.26 ± 5% -0.5 0.78 ± 6% perf-profile.self.cycles-pp.prepend_path
1.38 -0.4 0.94 perf-profile.self.cycles-pp.zap_pte_range
1.46 -0.4 1.02 perf-profile.self.cycles-pp.mas_prev_slot
1.60 -0.4 1.22 perf-profile.self.cycles-pp.mas_store_gfp
1.57 -0.4 1.21 perf-profile.self.cycles-pp.mas_next_slot
1.20 ± 5% -0.4 0.85 ± 3% perf-profile.self.cycles-pp.perf_iterate_sb
1.20 -0.3 0.88 perf-profile.self.cycles-pp.mas_find
1.42 -0.3 1.10 perf-profile.self.cycles-pp.memcpy_orig
1.06 -0.3 0.73 perf-profile.self.cycles-pp.vms_gather_munmap_vmas
0.84 ± 3% -0.3 0.53 ± 3% perf-profile.self.cycles-pp.kfree
0.83 ± 8% -0.3 0.53 ± 9% perf-profile.self.cycles-pp.shmem_get_unmapped_area
0.85 ± 3% -0.3 0.56 ± 2% perf-profile.self.cycles-pp.do_syscall_64
1.20 -0.3 0.91 perf-profile.self.cycles-pp.__zap_vma_range
0.63 -0.3 0.38 ± 2% perf-profile.self.cycles-pp.mas_preallocate
0.70 ± 2% -0.2 0.46 perf-profile.self.cycles-pp.unmapped_area_topdown
0.59 -0.2 0.34 ± 2% perf-profile.self.cycles-pp.__vma_start_exclude_readers
0.59 ± 2% -0.2 0.36 ± 3% perf-profile.self.cycles-pp.copy_from_kernel_nofault
0.48 -0.2 0.26 perf-profile.self.cycles-pp.__vm_munmap
0.58 -0.2 0.37 perf-profile.self.cycles-pp.ksys_mmap_pgoff
0.49 ± 8% -0.2 0.28 ± 5% perf-profile.self.cycles-pp.perf_event_mmap_output
0.54 ± 4% -0.2 0.33 ± 5% perf-profile.self.cycles-pp.queue_event
0.94 ± 4% -0.2 0.74 ± 3% perf-profile.self.cycles-pp.do_mmap
0.57 ± 3% -0.2 0.36 ± 3% perf-profile.self.cycles-pp.fget
0.51 -0.2 0.31 ± 2% perf-profile.self.cycles-pp.vma_merge_new_range
0.42 -0.2 0.22 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.47 ± 3% -0.2 0.27 ± 3% perf-profile.self.cycles-pp.x64_sys_call
0.58 -0.2 0.39 perf-profile.self.cycles-pp.up_write
0.38 -0.2 0.20 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.40 -0.2 0.23 ± 3% perf-profile.self.cycles-pp.security_vm_enough_memory_mm
0.91 ± 5% -0.2 0.74 perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.32 ± 2% -0.2 0.16 ± 7% perf-profile.self.cycles-pp.current_time
0.62 -0.2 0.46 perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown
0.62 ± 3% -0.2 0.46 perf-profile.self.cycles-pp.perf_event_mmap_event
0.42 ± 3% -0.2 0.26 ± 2% perf-profile.self.cycles-pp.vm_area_alloc
0.88 -0.2 0.73 perf-profile.self.cycles-pp.__mmap_new_vma
0.51 -0.2 0.36 perf-profile.self.cycles-pp.free_p4d_range
1.40 -0.2 1.25 perf-profile.self.cycles-pp.mas_rev_awalk
0.38 ± 3% -0.2 0.23 ± 2% perf-profile.self.cycles-pp.up_read
0.39 ± 2% -0.1 0.24 ± 3% perf-profile.self.cycles-pp.down_write
0.38 -0.1 0.23 ± 2% perf-profile.self.cycles-pp.fput
0.53 ± 2% -0.1 0.39 ± 5% perf-profile.self.cycles-pp.__mmap
0.44 ± 3% -0.1 0.31 ± 4% perf-profile.self.cycles-pp.d_path
0.64 ± 2% -0.1 0.51 ± 2% perf-profile.self.cycles-pp.kmem_cache_free
0.40 -0.1 0.27 ± 3% perf-profile.self.cycles-pp.__thp_vma_allowable_orders
0.34 -0.1 0.22 ± 2% perf-profile.self.cycles-pp.can_vma_merge_right
0.36 -0.1 0.22 ± 2% perf-profile.self.cycles-pp.vm_mmap_pgoff
0.32 ± 3% -0.1 0.20 ± 2% perf-profile.self.cycles-pp.unlink_file_vma_batch_process
0.58 -0.1 0.47 perf-profile.self.cycles-pp.do_vmi_align_munmap
0.22 -0.1 0.11 ± 3% perf-profile.self.cycles-pp.unlink_anon_vmas
0.34 ± 4% -0.1 0.23 ± 4% perf-profile.self.cycles-pp.touch_atime
0.28 ± 2% -0.1 0.16 ± 3% perf-profile.self.cycles-pp.mmap_region
0.33 ± 4% -0.1 0.22 ± 3% perf-profile.self.cycles-pp.strlen
0.21 ± 3% -0.1 0.10 ± 3% perf-profile.self.cycles-pp.vma_set_page_prot
0.35 ± 2% -0.1 0.24 ± 2% perf-profile.self.cycles-pp.may_expand_vm
0.33 ± 2% -0.1 0.22 ± 2% perf-profile.self.cycles-pp.pte_offset_map_lock
0.42 ± 5% -0.1 0.32 ± 3% perf-profile.self.cycles-pp.down_write_killable
0.28 -0.1 0.18 ± 2% perf-profile.self.cycles-pp.downgrade_write
0.32 ± 3% -0.1 0.22 ± 2% perf-profile.self.cycles-pp._raw_spin_lock
0.35 ± 5% -0.1 0.25 ± 12% perf-profile.self.cycles-pp.atime_needs_update
0.26 -0.1 0.15 ± 2% perf-profile.self.cycles-pp.cap_vm_enough_memory
0.30 -0.1 0.20 ± 2% perf-profile.self.cycles-pp.free_pgd_range
0.34 ± 2% -0.1 0.24 ± 7% perf-profile.self.cycles-pp.__munmap
0.24 -0.1 0.14 ± 3% perf-profile.self.cycles-pp.freader_init_from_file
0.24 ± 3% -0.1 0.14 ± 9% perf-profile.self.cycles-pp.static_key_count
0.35 ± 2% -0.1 0.25 ± 2% perf-profile.self.cycles-pp.__vma_start_write
0.35 ± 4% -0.1 0.25 ± 7% perf-profile.self.cycles-pp.vm_get_page_prot
0.31 -0.1 0.21 ± 2% perf-profile.self.cycles-pp.tlb_gather_mmu
0.28 ± 4% -0.1 0.18 ± 6% perf-profile.self.cycles-pp.uprobe_mmap
0.34 -0.1 0.25 ± 2% perf-profile.self.cycles-pp.mmap_action_complete
0.21 ± 2% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.prandom_u32_state
0.25 -0.1 0.16 ± 3% perf-profile.self.cycles-pp.unmap_region
0.68 ± 3% -0.1 0.59 ± 2% perf-profile.self.cycles-pp.vms_complete_munmap_vmas
1.12 ± 3% -0.1 1.03 ± 2% perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
0.33 -0.1 0.24 ± 2% perf-profile.self.cycles-pp.__build_id_parse
0.23 ± 3% -0.1 0.15 ± 11% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg
0.20 ± 4% -0.1 0.12 ± 8% perf-profile.self.cycles-pp.__mmap_complete
0.25 -0.1 0.17 ± 3% perf-profile.self.cycles-pp.filemap_get_entry
0.27 -0.1 0.20 ± 2% perf-profile.self.cycles-pp.__filemap_get_folio_mpol
0.22 ± 2% -0.1 0.15 ± 4% perf-profile.self.cycles-pp.vma_interval_tree_remove
0.09 ± 5% -0.1 0.02 ±100% perf-profile.self.cycles-pp.can_vma_merge_left
0.38 -0.1 0.31 perf-profile.self.cycles-pp.free_pgtables
0.19 ± 3% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.remove_vma
0.28 ± 2% -0.1 0.21 perf-profile.self.cycles-pp.tlb_finish_mmu
0.19 ± 3% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.cap_capable
0.31 ± 2% -0.1 0.25 ± 2% perf-profile.self.cycles-pp.userfaultfd_unmap_complete
0.18 ± 3% -0.1 0.12 ± 3% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.28 -0.1 0.22 ± 3% perf-profile.self.cycles-pp.perf_event_mmap
0.28 ± 2% -0.1 0.22 ± 3% perf-profile.self.cycles-pp.mas_prev
0.08 ± 7% -0.1 0.02 ±100% perf-profile.self.cycles-pp.flush_tlb_batched_pending
0.12 ± 3% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.unlink_file_vma_batch_add
0.15 ± 2% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.ksm_vma_flags
0.14 ± 3% -0.1 0.09 ± 3% perf-profile.self.cycles-pp.__vm_enough_memory
0.29 ± 2% -0.1 0.24 ± 2% perf-profile.self.cycles-pp.shmem_mmap_prepare
0.14 ± 4% -0.0 0.09 perf-profile.self.cycles-pp.vma_is_shared_writable
0.15 ± 2% -0.0 0.10 perf-profile.self.cycles-pp.__pte_offset_map
0.20 ± 2% -0.0 0.15 perf-profile.self.cycles-pp.mas_next_range
0.16 ± 3% -0.0 0.11 ± 5% perf-profile.self.cycles-pp.ima_file_mmap
0.19 ± 3% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.uprobe_munmap
0.14 ± 3% -0.0 0.09 ± 3% perf-profile.self.cycles-pp.mm_get_unmapped_area
0.33 -0.0 0.28 ± 2% perf-profile.self.cycles-pp.unmap_vmas
0.12 ± 5% -0.0 0.07 perf-profile.self.cycles-pp.__rb_insert_augmented
0.12 ± 2% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.__x64_sys_mmap
0.23 ± 2% -0.0 0.18 ± 3% perf-profile.self.cycles-pp.freader_get_folio
0.10 ± 4% -0.0 0.06 ± 5% perf-profile.self.cycles-pp.prepend_copy
0.27 ± 2% -0.0 0.22 ± 3% perf-profile.self.cycles-pp.do_vmi_munmap
0.19 ± 4% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.vma_is_special_huge
0.10 ± 5% -0.0 0.06 ± 5% perf-profile.self.cycles-pp.rb_next
0.09 ± 3% -0.0 0.06 ± 9% perf-profile.self.cycles-pp._raw_spin_unlock
0.10 ± 13% -0.0 0.06 ± 19% perf-profile.self.cycles-pp.memfd_check_seals_mmap
0.10 -0.0 0.07 ± 4% perf-profile.self.cycles-pp.mas_next_setup
0.19 ± 2% -0.0 0.16 ± 4% perf-profile.self.cycles-pp.copy_from_kernel_nofault_allowed
0.12 ± 2% -0.0 0.09 ± 3% perf-profile.self.cycles-pp.vm_unmapped_area
0.08 ± 5% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.prepend
0.23 -0.0 0.21 ± 2% perf-profile.self.cycles-pp.freader_fetch
0.24 -0.0 0.22 ± 2% perf-profile.self.cycles-pp.mas_wr_store_entry
0.21 ± 3% -0.0 0.19 ± 2% perf-profile.self.cycles-pp.mas_prev_setup
0.09 ± 4% -0.0 0.07 ± 4% perf-profile.self.cycles-pp.get_align_mask
0.08 ± 6% -0.0 0.06 perf-profile.self.cycles-pp.__x64_sys_munmap
0.07 ± 7% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.vm_area_free
0.12 ± 4% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.is_vmalloc_addr
0.05 ± 6% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.trylock_stock
0.06 ± 9% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.rcu_cblist_dequeue
0.13 +0.0 0.15 ± 4% perf-profile.self.cycles-pp.khugepaged_enter_vma
0.06 +0.0 0.09 ± 3% perf-profile.self.cycles-pp.__call_rcu_common
0.04 ± 57% +0.0 0.07 perf-profile.self.cycles-pp.mmap@plt
0.11 ± 4% +0.0 0.15 ± 4% perf-profile.self.cycles-pp.security_mmap_file
0.14 ± 7% +0.0 0.18 ± 7% perf-profile.self.cycles-pp.__refill_obj_stock
0.06 ± 5% +0.0 0.10 ± 3% perf-profile.self.cycles-pp.__pi_memcpy
0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.build_detached_freelist
0.60 ± 3% +0.1 0.66 ± 4% perf-profile.self.cycles-pp.__kfree_rcu_sheaf
0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.alloc_from_new_slab
0.03 ± 77% +0.1 0.10 ± 8% perf-profile.self.cycles-pp.xas_load
0.01 ±173% +0.1 0.11 ± 4% perf-profile.self.cycles-pp.barn_get_empty_sheaf
0.23 ± 4% +0.1 0.36 ± 2% perf-profile.self.cycles-pp.__account_obj_stock
0.32 ± 2% +0.2 0.48 perf-profile.self.cycles-pp.__rcu_free_sheaf_prepare
0.00 +0.2 0.17 ± 2% perf-profile.self.cycles-pp.mas_wr_walk_index
0.62 ± 3% +0.2 0.81 perf-profile.self.cycles-pp.mas_wr_walk_descend
0.08 ± 5% +0.2 0.27 ± 3% perf-profile.self.cycles-pp.setup_object
0.30 +0.3 0.55 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.53 +0.3 0.81 perf-profile.self.cycles-pp.mas_store_prealloc
0.00 +0.4 0.36 perf-profile.self.cycles-pp.kmem_cache_return_sheaf
0.00 +0.4 0.36 perf-profile.self.cycles-pp.mas_prev_node
0.00 +0.5 0.47 perf-profile.self.cycles-pp.mas_next_node
0.00 +0.5 0.52 perf-profile.self.cycles-pp.kmem_cache_alloc_from_sheaf_noprof
0.00 +0.6 0.60 perf-profile.self.cycles-pp.mas_ascend
0.00 +0.7 0.65 perf-profile.self.cycles-pp.multi_src_setup
0.00 +0.7 0.71 ± 4% perf-profile.self.cycles-pp.kmem_cache_prefill_sheaf
1.12 +0.8 1.95 perf-profile.self.cycles-pp.__refill_objects_node
1.18 +0.9 2.06 perf-profile.self.cycles-pp.__slab_free
0.64 +1.3 1.94 perf-profile.self.cycles-pp.__kmem_cache_free_bulk
0.00 +1.5 1.54 perf-profile.self.cycles-pp.node_finalise
0.00 +2.2 2.21 perf-profile.self.cycles-pp.node_copy
0.00 +2.7 2.66 perf-profile.self.cycles-pp.cp_data_write
0.00 +2.9 2.88 perf-profile.self.cycles-pp.mas_wr_split
0.50 ± 2% +2.9 3.41 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.00 +3.5 3.53 perf-profile.self.cycles-pp.mas_wr_spanning_store
0.00 +4.3 4.26 perf-profile.self.cycles-pp.dst_setup
0.00 +4.3 4.34 perf-profile.self.cycles-pp.mas_topiary_replace
>
> Let me know if it'd be better to provide a copy based off 280b792cac62d
> instead.
>
> Thanks,
> Liam
> From 98e443b159a75a6cc34af839c122a9e5eea2fe19 Mon Sep 17 00:00:00 2001
> From: "Liam R. Howlett" <[email protected]>
> Date: Fri, 15 May 2026 17:37:06 -0400
> Subject: [PATCH] maple_tree: Restore old tree layout using new scatter-gather
> node copy
>
> The prior bignode rebalance and split operations were greedy in getting
> data to the lower range nodes (often referred to as the left side of the
> tree). The new scatter-gather node copy was more fair on splitting
> data, but this lead to a regression with mmap2_processes.
>
> Restoring the greedy nature of getting data further left in the tree,
> when possible, turns out to perform better. Note that it's not entirely
> greedy and tries to leave some room for modifications to both nodes
> later, if possible.
>
> The decision of where to push the majority of the data is relayed as a
> 'hint' on which side to favour. Passing around a hint turns out to
> simplify a lot of logic that was otherwise derived from the siblings max
> or size.
>
> Signed-off-by: Liam R. Howlett (Oracle) <[email protected]>
> ---
> include/linux/maple_tree.h | 1 +
> lib/maple_tree.c | 427 ++++++++++++++++++++++++++-----------
> lib/test_maple_tree.c | 5 +-
> 3 files changed, 305 insertions(+), 128 deletions(-)
>
> diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
> index 4a5631906aff..c4c3b31614b0 100644
> --- a/include/linux/maple_tree.h
> +++ b/include/linux/maple_tree.h
> @@ -192,6 +192,7 @@ struct maple_copy {
> unsigned char d_count;
> unsigned char split;
> unsigned char data;
> + unsigned char write_start;
> unsigned char height;
> };
>
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index 60ae5e6fc1ee..b83a5b902e56 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -89,6 +89,10 @@
> */
> #define MA_STATE_PREALLOC 1
>
> +#define MAPLE_DST_NEUTRAL 0
> +#define MAPLE_DST_FAVOUR_LEFT 1
> +#define MAPLE_DST_FAVOUR_RIGHT 2
> +
> #define ma_parent_ptr(x) ((struct maple_pnode *)(x))
> #define mas_tree_parent(x) ((unsigned long)(x->tree) | MA_ROOT_PARENT)
> #define ma_mnode_ptr(x) ((struct maple_node *)(x))
> @@ -1720,21 +1724,26 @@ static inline void mas_wr_node_walk(struct ma_wr_state *wr_mas)
> wr_mas->offset_end = mas->offset = offset;
> }
>
> -static inline void rebalance_sib(struct ma_state *parent, struct ma_state *sib)
> +static inline unsigned char rebalance_sib(struct ma_state *parent,
> + struct ma_state *sib)
> {
> *sib = *parent;
> /* Prioritize move right to pull data left */
> - if (sib->offset < sib->end)
> + if (sib->offset < sib->end) {
> sib->offset++;
> - else
> - sib->offset--;
> + mas_descend(sib);
> + sib->end = mas_data_end(sib);
> + return MAPLE_DST_FAVOUR_RIGHT;
> + }
>
> + sib->offset--;
> mas_descend(sib);
> sib->end = mas_data_end(sib);
> + return MAPLE_DST_FAVOUR_LEFT;
> }
>
> static inline
> -void spanning_sib(struct ma_wr_state *l_wr_mas,
> +unsigned char spanning_sib(struct ma_wr_state *l_wr_mas,
> struct ma_wr_state *r_wr_mas, struct ma_state *nneighbour)
> {
> struct ma_state l_tmp = *l_wr_mas->mas;
> @@ -1754,7 +1763,7 @@ void spanning_sib(struct ma_wr_state *l_wr_mas,
>
> r_tmp.end = mas_data_end(&r_tmp);
> *nneighbour = r_tmp;
> - return;
> + return MAPLE_DST_FAVOUR_RIGHT;
> } else if (l_tmp.offset) {
> l_tmp.offset--;
> do {
> @@ -1764,11 +1773,13 @@ void spanning_sib(struct ma_wr_state *l_wr_mas,
>
> l_tmp.end = l_tmp.offset;
> *nneighbour = l_tmp;
> - return;
> + return MAPLE_DST_FAVOUR_LEFT;
> }
> } while (!mte_is_root(r_tmp.node));
>
> WARN_ON_ONCE(1);
> + nneighbour->end = 0;
> + return MAPLE_DST_NEUTRAL;
> }
>
> /*
> @@ -2151,74 +2162,147 @@ static inline void cp_data_calc(struct maple_copy *cp,
> cp->data += r_wr_mas->mas->end - r_wr_mas->offset_end;
> }
>
> -static bool data_fits(struct ma_state *sib, struct ma_state *mas,
> +#define MAPLE_TEST_ALT_SIB 3
> +#define MAPLE_NO_SPACE (-1)
> +#define MAPLE_MIN_SPARE_SLOTS 2
> +
> +static int data_fits(unsigned char end, struct ma_state *mas,
> struct maple_copy *cp)
> {
> unsigned char new_data;
> enum maple_type type;
> unsigned char space;
> - unsigned char end;
> + int spare_slots;
>
> + /*
> + * end is an index while cp->data/space are sizes, so a strict fit would
> + * allow new_data <= space. One spare slot (new_data < space) avoids full
> + * pack. Keep one more spare slot here to reduce split/rebalance churn.
> + */
> type = mte_node_type(mas->node);
> space = 2 * mt_slots[type];
> - end = sib->end;
> -
> new_data = end + 1 + cp->data;
> - if (new_data > space)
> - return false;
> -
> - /*
> - * This is off by one by design. The extra space is left to reduce
> - * jitter in operations that add then remove two entries.
> - *
> - * end is an index while new space and data are both sizes. Adding one
> - * to end to convert the index to a size means that the below
> - * calculation should be <=, but we want to keep an extra space in nodes
> - * to reduce jitter.
> - *
> - * Note that it is still possible to get a full node on the left by the
> - * NULL landing exactly on the split. The NULL ending of a node happens
> - * in the dst_setup() function, where we will either increase the split
> - * by one or decrease it by one, if possible. In the case of split
> - * (this case), it is always possible to shift the spilt by one - again
> - * because there is at least one slot free by the below checking.
> - */
> - if (new_data < space)
> - return true;
> + spare_slots = space - new_data;
> + if (spare_slots < MAPLE_MIN_SPARE_SLOTS)
> + return MAPLE_NO_SPACE;
>
> - return false;
> + return spare_slots;
> }
>
> -static inline void push_data_sib(struct maple_copy *cp, struct ma_state *mas,
> - struct ma_state *sib, struct ma_state *parent)
> +static inline int push_data_sib_left(struct maple_copy *cp,
> + struct ma_state *mas, struct ma_state *sib,
> + struct ma_state *parent)
> {
> -
> - if (mte_is_root(mas->node))
> - goto no_push;
> -
> + if (!parent->offset)
> + return MAPLE_NO_SPACE;
>
> *sib = *parent;
> - if (sib->offset) {
> - sib->offset--;
> - mas_descend(sib);
> - sib->end = mas_data_end(sib);
> - if (data_fits(sib, mas, cp)) /* Push left */
> - return;
> + sib->offset--;
> + mas_descend(sib);
> + sib->end = mas_data_end(sib);
>
> - *sib = *parent;
> - }
> + return data_fits(sib->end, mas, cp);
> +}
>
> - if (sib->offset >= sib->end)
> - goto no_push;
> +static inline int push_data_sib_right(struct maple_copy *cp,
> + struct ma_state *mas, struct ma_state *sib,
> + struct ma_state *parent)
> +{
> + if (parent->offset >= parent->end)
> + return MAPLE_NO_SPACE;
>
> + *sib = *parent;
> sib->offset++;
> mas_descend(sib);
> sib->end = mas_data_end(sib);
> - if (data_fits(sib, mas, cp)) /* Push right*/
> - return;
>
> -no_push:
> + return data_fits(sib->end, mas, cp);
> +}
> +
> +static inline unsigned char push_data_sib_split(struct maple_copy *cp,
> + struct ma_state *mas, struct ma_state *sib,
> + struct ma_state *parent)
> +{
> + struct ma_state left;
> + struct ma_state right;
> + int left_spare = MAPLE_NO_SPACE;
> + int right_spare = MAPLE_NO_SPACE;
> +
> + if (!mte_is_root(mas->node)) {
> + left_spare = push_data_sib_left(cp, mas, &left, parent);
> + if (left_spare >= MAPLE_TEST_ALT_SIB) {
> + *sib = left;
> + return MAPLE_DST_FAVOUR_LEFT;
> + }
> +
> + right_spare = push_data_sib_right(cp, mas, &right, parent);
> +
> + if ((left_spare >= 0) && (right_spare >= 0)) {
> + if (left.end <= right.end) {
> + *sib = left;
> + return MAPLE_DST_FAVOUR_LEFT;
> + }
> +
> + *sib = right;
> + return MAPLE_DST_FAVOUR_RIGHT;
> + }
> +
> + if (left_spare >= 0) {
> + *sib = left;
> + return MAPLE_DST_FAVOUR_LEFT;
> + }
> +
> + if (right_spare >= 0) {
> + *sib = right;
> + return MAPLE_DST_FAVOUR_RIGHT;
> + }
> + }
> +
> sib->end = 0;
> + return MAPLE_DST_NEUTRAL;
> +}
> +
> +static inline unsigned char push_data_sib_rebalance(struct maple_copy *cp,
> + struct ma_state *mas, struct ma_state *sib,
> + struct ma_state *parent)
> +{
> + struct ma_state left;
> + struct ma_state right;
> + int left_spare = MAPLE_NO_SPACE;
> + int right_spare = MAPLE_NO_SPACE;
> +
> + if (!mte_is_root(mas->node)) {
> + right_spare = push_data_sib_right(cp, mas, &right, parent);
> + if (right_spare >= MAPLE_TEST_ALT_SIB) {
> + *sib = right;
> + return MAPLE_DST_FAVOUR_RIGHT;
> + }
> +
> + left_spare = push_data_sib_left(cp, mas, &left, parent);
> +
> + if ((left_spare >= 0) && (right_spare >= 0)) {
> + if (left.end < right.end) {
> + *sib = left;
> + return MAPLE_DST_FAVOUR_LEFT;
> + }
> +
> + *sib = right;
> + return MAPLE_DST_FAVOUR_RIGHT;
> + }
> +
> + if (right_spare >= 0) {
> + *sib = right;
> + return MAPLE_DST_FAVOUR_RIGHT;
> + }
> +
> + if (left_spare >= 0) {
> + *sib = left;
> + return MAPLE_DST_FAVOUR_LEFT;
> + }
> + }
> +
> + sib->end = 0;
> + return MAPLE_DST_NEUTRAL;
> }
>
> /*
> @@ -2232,29 +2316,27 @@ static inline void push_data_sib(struct maple_copy *cp, struct ma_state *mas,
> * indicate it will not be used.
> *
> */
> -static inline void rebalance_data(struct maple_copy *cp,
> +static inline unsigned char rebalance_data(struct maple_copy *cp,
> struct ma_wr_state *wr_mas, struct ma_state *sib,
> struct ma_state *parent)
> {
> + unsigned char hint = MAPLE_DST_NEUTRAL;
> +
> cp_data_calc(cp, wr_mas, wr_mas);
> sib->end = 0;
> +
> if (cp->data > mt_slots[wr_mas->type]) {
> - push_data_sib(cp, wr_mas->mas, sib, parent);
> - if (sib->end)
> - goto use_sib;
> + hint = push_data_sib_rebalance(cp, wr_mas->mas, sib, parent);
> } else if (cp->data <= mt_min_slots[wr_mas->type]) {
> if ((wr_mas->mas->min != 0) ||
> - (wr_mas->mas->max != ULONG_MAX)) {
> - rebalance_sib(parent, sib);
> - goto use_sib;
> - }
> + (wr_mas->mas->max != ULONG_MAX))
> + hint = rebalance_sib(parent, sib);
> }
>
> - return;
> -
> -use_sib:
> + if (hint != MAPLE_DST_NEUTRAL)
> + cp->data += sib->end + 1;
>
> - cp->data += sib->end + 1;
> + return hint;
> }
>
> /*
> @@ -2267,18 +2349,64 @@ static inline void rebalance_data(struct maple_copy *cp,
> * Note: @cp->data is a size and not indexed by 0. @sib->end may be set to 0 to
> * indicate it will not be used.
> */
> -static inline void spanning_data(struct maple_copy *cp,
> +static inline unsigned char spanning_data(struct maple_copy *cp,
> struct ma_wr_state *l_wr_mas, struct ma_wr_state *r_wr_mas,
> struct ma_state *sib)
> {
> + unsigned char hint = MAPLE_DST_NEUTRAL;
> +
> cp_data_calc(cp, l_wr_mas, r_wr_mas);
> if (((l_wr_mas->mas->min != 0) || (r_wr_mas->mas->max != ULONG_MAX)) &&
> (cp->data <= mt_min_slots[l_wr_mas->type])) {
> - spanning_sib(l_wr_mas, r_wr_mas, sib);
> - cp->data += sib->end + 1;
> + hint = spanning_sib(l_wr_mas, r_wr_mas, sib);
> + if (hint != MAPLE_DST_NEUTRAL)
> + cp->data += sib->end + 1;
> } else {
> sib->end = 0;
> }
> +
> + return hint;
> +}
> +
> +static inline unsigned char cp_hint_split(struct maple_copy *cp,
> + enum maple_type mt, unsigned char hint)
> +{
> + unsigned char min_split;
> + unsigned char max_split;
> + unsigned int target = cp->split;
> +
> + /*
> + * Keep the original split bounds: these preserve minimum occupancy and
> + * leave the one-slot margin used by the NULL-end adjustment in
> + * cp_data_write().
> + */
> + max_split = mt_slots[mt] - 1;
> + if (cp->data - mt_min_slots[mt] - 2 < max_split)
> + max_split = cp->data - mt_min_slots[mt] - 2;
> +
> + min_split = mt_min_slots[mt];
> + if (cp->data - mt_slots[mt] > min_split)
> + min_split = cp->data - mt_slots[mt];
> +
> + switch (hint) {
> + case MAPLE_DST_FAVOUR_LEFT:
> + target = mt_slots[mt] - 2;
> + break;
> + case MAPLE_DST_FAVOUR_RIGHT:
> + target = min_split;
> + break;
> + default:
> + break;
> + }
> +
> + if (target < min_split)
> + target = min_split;
> + if (target > max_split)
> + target = max_split;
> +
> + cp->split = target;
> +
> + return cp->split;
> }
>
> /*
> @@ -2286,9 +2414,11 @@ static inline void spanning_data(struct maple_copy *cp,
> * @cp: The maple copy node
> * @mas: The maple state
> * @mt: The source node type
> + * @hint: Split preference from write placement and sibling side
> */
> static inline
> -void dst_setup(struct maple_copy *cp, struct ma_state *mas, enum maple_type mt)
> +void dst_setup(struct maple_copy *cp, struct ma_state *mas, enum maple_type mt,
> + unsigned char hint)
> {
> /* Data is 1 indexed, every src has +1 added. */
>
> @@ -2300,8 +2430,10 @@ void dst_setup(struct maple_copy *cp, struct ma_state *mas, enum maple_type mt)
>
> cp->split = (cp->data - 1) / 2;
> cp->d_count = 2;
> - if (cp->data < mt_slots[mt] * 2)
> + if (cp->data < mt_slots[mt] * 2) {
> + cp_hint_split(cp, mt, hint);
> goto node_setup;
> + }
>
> if (cp->data == mt_slots[mt] * 2) {
> unsigned char off;
> @@ -2407,16 +2539,23 @@ static inline void init_cp_src(struct maple_copy *cp)
> * @l_wr_mas: The left write maple state
> * @r_wr_mas: The right write maple state
> * @sib: The sibling maple state
> + * @hint: The sibling direction hint
> *
> - * Note: @sib->end == 0 indicates no sibling will be used.
> + * Note: @hint == MAPLE_DST_NEUTRAL indicates no sibling will be used.
> */
> static inline
> void multi_src_setup(struct maple_copy *cp, struct ma_wr_state *l_wr_mas,
> - struct ma_wr_state *r_wr_mas, struct ma_state *sib)
> + struct ma_wr_state *r_wr_mas, struct ma_state *sib,
> + unsigned char hint)
> {
> + unsigned char offset = 0;
> +
> cp->s_count = 0;
> - if (sib->end && sib->max < l_wr_mas->mas->min)
> +
> + if (hint == MAPLE_DST_FAVOUR_LEFT) {
> append_mas_cp(cp, sib, 0, sib->end);
> + offset += cp->src[cp->s_count - 1].end + 1;
> + }
>
> /* Copy left 0 - offset */
> if (l_wr_mas->mas->offset) {
> @@ -2424,8 +2563,10 @@ void multi_src_setup(struct maple_copy *cp, struct ma_wr_state *l_wr_mas,
>
> append_wr_mas_cp(cp, l_wr_mas, 0, off);
> cp->src[cp->s_count - 1].max = cp->min - 1;
> + offset += off + 1;
> }
>
> + cp->write_start = offset;
> init_cp_src(cp);
>
> /* Copy right either from offset or offset + 1 pending on r_max */
> @@ -2433,7 +2574,7 @@ void multi_src_setup(struct maple_copy *cp, struct ma_wr_state *l_wr_mas,
> append_wr_mas_cp(cp, r_wr_mas, r_wr_mas->offset_end + 1,
> r_wr_mas->mas->end);
>
> - if (sib->end && sib->min > r_wr_mas->mas->max)
> + if (hint == MAPLE_DST_FAVOUR_RIGHT)
> append_mas_cp(cp, sib, 0, sib->end);
> }
>
> @@ -2506,21 +2647,51 @@ void cp_data_write(struct maple_copy *cp, struct ma_state *mas)
> /* Handle null entries */
> if (cp->dst[d].max != ULONG_MAX &&
> !ma_slots(dst, d_mt)[dst_offset - 1]) {
> - if (s_offset == cp->src[s].start) {
> - s--;
> - src = cp->src[s].node;
> - src_end = cp->src[s].end;
> - s_max = cp->src[s].max;
> - s_mt = cp->src[s].mt;
> - s_offset = src_end;
> + if ((dst_offset < mt_slots[d_mt]) &&
> + (cp->data - data_offset > mt_min_slots[d_mt] + 1)) {
> + d_max = node_copy(mas, src, s_offset, 1, s_max, s_mt,
> + dst, dst_offset, d_mt);
> + dst_offset++;
> + data_offset++;
> + s_offset++;
> + if (s_offset > src_end) {
> + s++;
> + if (s >= cp->s_count) {
> + cp->dst[d].max = d_max;
> + node_finalise(dst, d_mt,
> + dst_offset);
> + return;
> + }
> +
> + src = cp->src[s].node;
> + s_offset = cp->src[s].start;
> + src_end = cp->src[s].end;
> + s_max = cp->src[s].max;
> + s_mt = cp->src[s].mt;
> + }
> +
> + cp->dst[d].max = d_max;
> + if (WARN_ON_ONCE(!split))
> + split = 0;
> + else
> + split--;
> } else {
> - s_offset--;
> + if (s_offset == cp->src[s].start) {
> + s--;
> + src = cp->src[s].node;
> + src_end = cp->src[s].end;
> + s_max = cp->src[s].max;
> + s_mt = cp->src[s].mt;
> + s_offset = src_end;
> + } else {
> + s_offset--;
> + }
> + /* Set dst max and clear pivot */
> + split++;
> + data_offset--;
> + dst_offset--;
> + cp->dst[d].max = ma_pivots(dst, d_mt)[dst_offset - 1];
> }
> - /* Set dst max and clear pivot */
> - split++;
> - data_offset--;
> - dst_offset--;
> - cp->dst[d].max = ma_pivots(dst, d_mt)[dst_offset - 1];
> }
>
> node_finalise(dst, d_mt, dst_offset);
> @@ -2600,7 +2771,7 @@ static inline bool cp_is_new_root(struct maple_copy *cp, struct ma_state *mas)
>
> cp->data = cp->d_count;
> cp->s_count = 0;
> - dst_setup(cp, mas, mt);
> + dst_setup(cp, mas, mt, MAPLE_DST_NEUTRAL);
> init_cp_src(cp);
> node_copy(mas, cp->src[0].node, 0, cp->data, cp->max, maple_copy,
> cp->dst[0].node, 0, mt);
> @@ -2625,9 +2796,9 @@ static inline bool cp_is_new_root(struct maple_copy *cp, struct ma_state *mas)
> }
>
> static inline bool cp_converged(struct maple_copy *cp, struct ma_state *mas,
> - struct ma_state *sib)
> + unsigned char hint)
> {
> - if (cp->d_count != 1 || sib->end)
> + if (cp->d_count != 1 || hint != MAPLE_DST_NEUTRAL)
> return false;
>
> cp->dst[0].node->parent = ma_parent_ptr(mas_mn(mas)->parent);
> @@ -2646,14 +2817,12 @@ static inline bool cp_converged(struct maple_copy *cp, struct ma_state *mas,
> */
> static bool spanning_ascend(struct maple_copy *cp, struct ma_state *mas,
> struct ma_wr_state *l_wr_mas, struct ma_wr_state *r_wr_mas,
> - struct ma_state *sib)
> + struct ma_state *sib, unsigned char hint)
> {
> - if (sib->end) {
> - if (sib->max < l_wr_mas->mas->min)
> - *l_wr_mas->mas = *sib;
> - else
> - *r_wr_mas->mas = *sib;
> - }
> + if (hint == MAPLE_DST_FAVOUR_LEFT)
> + *l_wr_mas->mas = *sib;
> + else if (hint == MAPLE_DST_FAVOUR_RIGHT)
> + *r_wr_mas->mas = *sib;
>
> cp_dst_to_slots(cp, l_wr_mas->mas->min, r_wr_mas->mas->max, mas);
> if (cp_is_new_root(cp, mas))
> @@ -2692,16 +2861,16 @@ void copy_tree_location(const struct ma_state *src, struct ma_state *dst)
> */
> static inline bool rebalance_ascend(struct maple_copy *cp,
> struct ma_wr_state *wr_mas, struct ma_state *sib,
> - struct ma_state *parent)
> + struct ma_state *parent, unsigned char hint)
> {
> struct ma_state *mas;
> unsigned long min, max;
>
> mas = wr_mas->mas;
> - if (!sib->end) {
> + if (hint == MAPLE_DST_NEUTRAL) {
> min = mas->min;
> max = mas->max;
> - } else if (sib->min > mas->max) { /* Move right succeeded */
> + } else if (hint == MAPLE_DST_FAVOUR_RIGHT) {
> min = mas->min;
> max = sib->max;
> wr_mas->offset_end = parent->offset + 1;
> @@ -2716,7 +2885,7 @@ static inline bool rebalance_ascend(struct maple_copy *cp,
> if (cp_is_new_root(cp, mas))
> return false;
>
> - if (cp_converged(cp, mas, sib))
> + if (cp_converged(cp, mas, hint))
> return false;
>
> cp->height++;
> @@ -3049,6 +3218,7 @@ static void mas_wr_spanning_store(struct ma_wr_state *wr_mas)
> struct maple_copy cp;
> struct ma_state *mas;
> struct ma_state sib;
> + unsigned char hint;
>
> /* Left and Right side of spanning store */
> MA_STATE(r_mas, NULL, 0, 0);
> @@ -3107,11 +3277,11 @@ static void mas_wr_spanning_store(struct ma_wr_state *wr_mas)
>
> cp_leaf_init(&cp, mas, wr_mas, &r_wr_mas);
> do {
> - spanning_data(&cp, wr_mas, &r_wr_mas, &sib);
> - multi_src_setup(&cp, wr_mas, &r_wr_mas, &sib);
> - dst_setup(&cp, mas, wr_mas->type);
> + hint = spanning_data(&cp, wr_mas, &r_wr_mas, &sib);
> + multi_src_setup(&cp, wr_mas, &r_wr_mas, &sib, hint);
> + dst_setup(&cp, mas, wr_mas->type, hint);
> cp_data_write(&cp, mas);
> - } while (spanning_ascend(&cp, mas, wr_mas, &r_wr_mas, &sib));
> + } while (spanning_ascend(&cp, mas, wr_mas, &r_wr_mas, &sib, hint));
>
> mas_wmb_replace(mas, &cp);
> }
> @@ -3377,7 +3547,7 @@ static inline void mas_wr_append(struct ma_wr_state *wr_mas)
> */
> static inline bool split_ascend(struct maple_copy *cp,
> struct ma_wr_state *wr_mas, struct ma_state *sib,
> - struct ma_state *parent)
> + struct ma_state *parent, unsigned char hint)
> {
> struct ma_state *mas;
> unsigned long min, max;
> @@ -3386,21 +3556,19 @@ static inline bool split_ascend(struct maple_copy *cp,
> min = mas->min; /* push right, or normal split */
> max = mas->max;
> wr_mas->offset_end = parent->offset;
> - if (sib->end) {
> - if (sib->max < mas->min) {
> - min = sib->min; /* push left */
> - parent->offset--;
> - } else {
> - max = sib->max; /* push right */
> - wr_mas->offset_end++;
> - }
> + if (hint == MAPLE_DST_FAVOUR_LEFT) {
> + min = sib->min; /* push left */
> + parent->offset--;
> + } else if (hint == MAPLE_DST_FAVOUR_RIGHT) {
> + max = sib->max; /* push right */
> + wr_mas->offset_end++;
> }
>
> cp_dst_to_slots(cp, min, max, mas);
> if (cp_is_new_root(cp, mas))
> return false;
>
> - if (cp_converged(cp, mas, sib))
> + if (cp_converged(cp, mas, hint))
> return false;
>
> cp->height++;
> @@ -3420,19 +3588,24 @@ static inline bool split_ascend(struct maple_copy *cp,
> * indicate it will not be used.
> *
> */
> -static inline void split_data(struct maple_copy *cp,
> +static inline unsigned char split_data(struct maple_copy *cp,
> struct ma_wr_state *wr_mas, struct ma_state *sib,
> struct ma_state *parent)
> {
> + unsigned char hint = MAPLE_DST_NEUTRAL;
> +
> cp_data_calc(cp, wr_mas, wr_mas);
> +
> if (cp->data <= mt_slots[wr_mas->type]) {
> sib->end = 0;
> - return;
> + return hint;
> }
>
> - push_data_sib(cp, wr_mas->mas, sib, parent);
> - if (sib->end)
> + hint = push_data_sib_split(cp, wr_mas->mas, sib, parent);
> + if (hint != MAPLE_DST_NEUTRAL)
> cp->data += sib->end + 1;
> +
> + return hint;
> }
>
> /*
> @@ -3445,6 +3618,7 @@ static void mas_wr_split(struct ma_wr_state *wr_mas)
> struct ma_state *mas;
> struct maple_copy cp;
> struct ma_state sib;
> + unsigned char hint;
>
> mas = wr_mas->mas;
> trace_ma_write(TP_FCT, wr_mas->mas, 0, wr_mas->entry);
> @@ -3455,11 +3629,11 @@ static void mas_wr_split(struct ma_wr_state *wr_mas)
> mas_ascend(&parent);
> parent.end = mas_data_end(&parent);
> }
> - split_data(&cp, wr_mas, &sib, &parent);
> - multi_src_setup(&cp, wr_mas, wr_mas, &sib);
> - dst_setup(&cp, mas, wr_mas->type);
> + hint = split_data(&cp, wr_mas, &sib, &parent);
> + multi_src_setup(&cp, wr_mas, wr_mas, &sib, hint);
> + dst_setup(&cp, mas, wr_mas->type, hint);
> cp_data_write(&cp, mas);
> - } while (split_ascend(&cp, wr_mas, &sib, &parent));
> + } while (split_ascend(&cp, wr_mas, &sib, &parent, hint));
>
> mas_wmb_replace(mas, &cp);
> }
> @@ -3478,6 +3652,7 @@ static void mas_wr_rebalance(struct ma_wr_state *wr_mas)
> struct ma_state *mas;
> struct maple_copy cp;
> struct ma_state sib;
> + unsigned char hint;
>
> /*
> * Rebalancing occurs if a node is insufficient. Data is rebalanced
> @@ -3498,11 +3673,11 @@ static void mas_wr_rebalance(struct ma_wr_state *wr_mas)
> mas_ascend(&parent);
> parent.end = mas_data_end(&parent);
> }
> - rebalance_data(&cp, wr_mas, &sib, &parent);
> - multi_src_setup(&cp, wr_mas, wr_mas, &sib);
> - dst_setup(&cp, mas, wr_mas->type);
> + hint = rebalance_data(&cp, wr_mas, &sib, &parent);
> + multi_src_setup(&cp, wr_mas, wr_mas, &sib, hint);
> + dst_setup(&cp, mas, wr_mas->type, hint);
> cp_data_write(&cp, mas);
> - } while (rebalance_ascend(&cp, wr_mas, &sib, &parent));
> + } while (rebalance_ascend(&cp, wr_mas, &sib, &parent, hint));
>
> mas_wmb_replace(mas, &cp);
> }
> diff --git a/lib/test_maple_tree.c b/lib/test_maple_tree.c
> index b9367c61e8b5..4033d1b85fee 100644
> --- a/lib/test_maple_tree.c
> +++ b/lib/test_maple_tree.c
> @@ -1723,9 +1723,10 @@ static noinline void __init check_gap_combining(struct maple_tree *mt)
> mn1 = mas.node;
> entry = mas_next(&mas, ULONG_MAX);
> MT_BUG_ON(mt, entry != xa_mk_value(index + 4));
> - mas_next(&mas, ULONG_MAX); /* go to the next entry. */
> mn2 = mas.node;
> - MT_BUG_ON(mt, mn1 == mn2); /* test the next entry is in the next node. */
> + MT_BUG_ON(mt, mn1 == mn2);
> + entry = mas_next(&mas, ULONG_MAX); /* go to the next entry. */
> + MT_BUG_ON(mt, entry != xa_mk_value(index + 5));
>
> /*
> * At this point, there is a gap of 3 at seq100[6]. Find it by
> --
> 2.47.3
>
config-7.1.0-rc4+
(text/plain, 246.5 KB) - not displayed