Re: [PATCH 0/4] mm/mempolicy: introduce package-aware weighted interleave

On Thu,  6 Aug 2026 17:09:31 +0900 Rakie Kim <[email protected]> wrote:

> Package-aware weighted interleave places a task's weighted-interleave
> pages on the NUMA nodes of its local package, so that interleave traffic
> does not have to cross the interconnect to another package. This keeps
> each node's weight aligned with the bandwidth the task actually gets
> from it, so effective bandwidth holds up on a system that has more than
> one package. (A package is a CPU socket together with the memory
> attached to it.)
> 
> Changes from RFC:
> https://lore.kernel.org/all/[email protected]/
> - Added an opt-in sysfs toggle (off by default) and a read-only sysfs
>   view of the package topology
> - Added topology validation with a clean fallback to plain weighted
>   interleave on unsupported topologies
> - Hardened the allocation, device-teardown, and node-hotplug paths

Hello Rakie,

I hope you are doing well! Sorry for the late repsonse.

I have a few thoughts, some of which are carry-overs from the RFC
discussion we had before! I think there are still some open questions,
and I wanted to get your opinion on some of them.

My first question is whether we want cross-socket allocations at all.
The examples you gave seem to line up with node-restricted interleave,
as opposed to cross-socket interleave. I think the wording that you
use to describe the feature in 4/4 (which I will copy below)

> The resolved mask is by construction a subset of the policy nodemask, which
> mempolicy already restricts to the task's cpuset; package mode can only
> narrow that set, never widen it, so cpusets and the task nodemask remain
> authoritative.

is 100% the right way to treat these package-aware (socket-aware)
interleaving allocations, but the example below

[...snip...]

> Applied the same way to every source, these weights give the map:
> 
>               node0  node1  node2  node3
> global:         2      2      1      1

[...snip...]

>               node0  node1  node2  node3
> from CPU 0:     2      0      1      0
> from CPU 1:     0      2      0      1

Is essentially the existing weighted interleave mechanism with a 
nodemask/cpuset applied. With that said, I think a more interesting and
illustrative example would be if the user truly would want to allow some
allocations to go through cross-socket, but be able to control the
ratio at which these slip through.

              node0  node1  node2  node3
from CPU 0:     3      1      2      0
from CPU 1:     0      3      1      2

Maybe even more illustrative of the true capabilities of this series
would be if you have an asymmetric system where you bind some
host-level monitoring / logging workloads to one node (say, node0) and
want that to be able to cross through to the other socket, but not the
other way around:

              node0  node1  node2  node3
from CPU 0:     3      1      2      0
from CPU 1:     0      2      0      1

Anyways, these are just super hypothetical scenarios and I don't even
know if the configuration that I'm listing would really be beneficial
for the system. I think that coming up with some illustrative usecases
which are now made possible by this series could help motivate why we
would want to interleave across sockets. 

> A task on CPU 0 now places pages on node0 (weight 2) and node2
> (weight 1) at 2:1, which matches their effective bandwidth of 300:150;
> a task on CPU 1 places on node1 and node3 the same way. Placement
> follows the bandwidth each task actually sees, NUMA locality is
> preserved, and interleave traffic stays off the interconnect.
> 
> To make this possible the kernel needs a notion of which nodes share a
> package. The NUMA distance model offers only relative latencies and no
> structural grouping, which is especially limiting for CXL memory nodes
> that come online without an explicit package association.
> 
> The series adds a package-aware topology layer that groups CPU and
> memory-only nodes into a "memory package", built from the physical
> package ids firmware reports and, for a memory-only node, an initiator
> CPU node or SLIT distances. A package can contain more than one CPU node
> or more than one memory-only node, so the layer maps a package to a set
> of nodes rather than to a single node or a single CXL device.
> 
> The feature is off by default and opt-in through a sysfs toggle. The
> package topology itself is exposed read-only under
> /sys/devices/system/package/; there is deliberately no writable
> override, since a machine whose firmware describes its topology
> incorrectly should be fixed in firmware. On a topology that does not
> have the symmetric shape the placement relies on, enabling is refused
> and any active mode degrades cleanly to the original flat behavior.

I was also hoping to see what this interface looks like and maybe
discuss how we should relay the information to the users, since this
seems to be a new addition from the RFC.

> Measured results:
> 
> System Configuration:
> - Processor: Dual-Socket Intel Xeon 6980P (Granite Rapids)

I think a description of this system's topology would help me understand
the results below a bit better : -)

> 1) Throughput (System Bandwidth)
>    - DRAM Only: 966 GB/s
>    - Weighted Interleave: 903 GB/s (7% decrease compared to DRAM Only)
>    - Package-Aware Weighted Interleave: 1329 GB/s (1.33 TB/s)
>      (38% increase compared to DRAM Only,
>       47% increase compared to Weighted Interleave)
> 
> 2) Loaded Latency (Under High Bandwidth)
>    - DRAM Only: 544 ns
>    - Weighted Interleave: 545 ns
>    - Package-Aware Weighted Interleave: 436 ns
>      (20% reduction compared to both)

Really awesome results!

> A small CXL driver change registers a CXL memory node into its package
> as the node comes online, using the initiator the driver resolves for
> the region; this is where the package layer gets the CPU-side
> association that plain NUMA distance does not carry.
> 
> The memory_package layer offers a broader interface for grouping and
> querying package topology - usable by memory tiering as well - and
> package-aware weighted interleave uses the subset it needs.

I was hoping you could expand on this a bit more. Aside from the
alloction-time placement strategy, did you have other ideas in mind for
who could ingest the package information to make tiering decisions?

I definitely think this series makes a lot of sense and I am
hoping to hear more about it. Thank you, I hope you have a great day!

Joshua