Re: [PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

On 2026-08-04 05:52, Donet Tom wrote:
> This series fixes topology device creation and proximity domain mappings in
> AMDKFD when a system does not provide a CRAT table and the driver generates a
> Virtual CRAT (VCRAT).
>
> The current implementation assumes that CPU NUMA node IDs are contiguous and
> that every NUMA node contains CPUs. During VCRAT generation, CPU topology
> entries and proximity domains are created only for NUMA nodes that have CPUs.
> GPU proximity domains are then allocated immediately after the CPU proximity
> domains, and the GPU I/O link (proximity_domain_to) is initialized using the
> NUMA node ID to which the GPU is attached, implicitly assuming that NUMA node
> IDs and proximity domains have a one-to-one mapping.
>
> These assumptions break on systems with:
>
> Sparse (non-contiguous) NUMA node IDs
> CPU-less NUMA nodes
> CPU-less and memory-less NUMA nodes
>
> For example:
>
> available: 3 nodes (0,2-3)
>
> node 0: CPUs present
> node 2: CPU-less
> node 3: CPU-less
>
> In this case, the driver creates a CPU topology device only for node 0 and
> assigns a single CPU proximity domain (0). However, the GPU VCRAT still
> references the NUMA node ID to which the GPU is attached (for example, node 3)
> in the proximity_domain_to field. Since no corresponding CPU proximity domain
> exists for node 3, the parser cannot find a matching proximity domain during
> VCRAT parsing, causing topology initialization to fail.

Hi Tom,

Thank you for the explanation and the patch series. I may have some gaps 
in my understanding that I would like to clarify. In my mind, I was 
using "proximity domain" and "NUMA node" interchangeably. You're 
demonstrating that they are not the same thing. Is that just a different 
way of labeling the same thing, or are NUMA nodes and proximity domains 
fundamentally different concepts.

Your code in patch 5 (kfd_proximity_domain_to_numa_node and 
kfd_numa_node_to_proximity_domain) seems to imply that there is, in 
fact, a 1:1 mapping, as it assumes that there is a unique translation in 
both directions. Am I missing something?

If they are just different numbering systems, do we really need to keep 
track of both proximity domains and NUMA nodes in the KFD topology? Or 
would it be sufficient to only track NUMA nodes, if that's what we 
really care about in the uAPI (KFD sysfs)?

Thanks,
   Felix


>
> The failure is observed as:
>
> amdgpu: Virtual CRAT table created for GPU
> amdgpu: Error parsing VCRAT
> kfd: amdgpu: Error adding device to topology
> kfd: amdgpu: Error initializing KFD node
>
>
> Since every online NUMA node is a valid topology object and can contain
> CPUs, memory, I/O links, or any combination of these, topology devices
> and proximity domains should be created for every online NUMA node
> rather than only for NUMA nodes that contain CPUs.
>
> To address this, this series introduces a new VCRAT subtype that
> records the mapping between the NUMA node ID and the generated VCRAT
> proximity domain. When topology devices are created, this information
> is stored in the corresponding topology device, allowing the driver to
> translate a NUMA node ID into its associated proximity domain whenever
> required.
>
> Returning to the previous example, the system contains three online
> NUMA nodes, so three CPU topology devices and three proximity domains
> are created, even though only one NUMA node contains CPUs. The NUMA
> node ID is stored in each topology device together with its generated
> proximity domain.
>
> Later, when the GPU VCRAT is generated, the driver only knows the NUMA
> node ID to which the GPU is attached (for example, node 3). Instead of
> assuming that the NUMA node ID is equal to the proximity domain, the
> driver walks the existing topology devices to locate the corresponding
> NUMA node and retrieves its generated proximity domain. In this
> example, NUMA node 3 maps to proximity domain 2, so
> proximity_domain_to is populated with the correct value.
>
> Since the GPU I/O link now references a valid proximity domain, VCRAT
> parsing completes successfully and topology initialization proceeds
> without errors on systems with sparse NUMA node IDs, CPU-less NUMA
> nodes, and CPU-less/memory-less NUMA nodes.
>
> This series consists of the following patches:
>
> Patch 1 removes an unused argument from
> kfd_create_crat_image_virtual() as a preparatory cleanup.
>
> Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
> NUMA node ID and its corresponding proximity domain.
>
> Patch 3 populates the NUMA affinity entries during VCRAT generation
> for every online NUMA node.
>
> Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
> NUMA node ID and proximity domain in the corresponding topology
> device.
>
> Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
> GPU's NUMA node ID to the corresponding CPU proximity domain before
> programming proximity_domain_to.
>
> Patch 6 creates proximity domains and VCRAT entries for all online
> NUMA nodes, including CPU-less and memory-less nodes.
>
> Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
> devices, exposing the associated NUMA node through the topology sysfs
> interface.
>
> Please note that the changes in this series are on a best effort basis from our
> end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the
> HW & SW stack) to kindly help with the review and provide feedback / comments on
> these patches
>
> Donet Tom (7):
>    drm/amdgpu: Remove unused argument from kfd_create_crat_image_virtual
>    drm/amdgpu: Add VCRAT NUMA affinity entry
>    drm/amdgpu: Populate NUMA affinity entries in VCRAT
>    drm/amdgpu: Parse NUMA affinity entries from VCRAT
>    drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
>    drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
>      and memory-less NUMA nodes
>    drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
>
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 +++++++++++++++-------
>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
>   drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
>   4 files changed, 168 insertions(+), 56 deletions(-)
>