Re: [PATCH 0/7] drm/amdgpu: Fix topology device creation and proximity domain mappings for sparse and CPU-less NUMA systems

On 2026-08-05 03:09, Donet Tom wrote:
>
> On 8/5/26 3:57 AM, Felix Kuehling wrote:
>>
>> On 2026-08-04 05:52, Donet Tom wrote:
>>> This series fixes topology device creation and proximity domain 
>>> mappings in
>>> AMDKFD when a system does not provide a CRAT table and the driver 
>>> generates a
>>> Virtual CRAT (VCRAT).
>>>
>>> The current implementation assumes that CPU NUMA node IDs are 
>>> contiguous and
>>> that every NUMA node contains CPUs. During VCRAT generation, CPU 
>>> topology
>>> entries and proximity domains are created only for NUMA nodes that 
>>> have CPUs.
>>> GPU proximity domains are then allocated immediately after the CPU 
>>> proximity
>>> domains, and the GPU I/O link (proximity_domain_to) is initialized 
>>> using the
>>> NUMA node ID to which the GPU is attached, implicitly assuming that 
>>> NUMA node
>>> IDs and proximity domains have a one-to-one mapping.
>>>
>>> These assumptions break on systems with:
>>>
>>> Sparse (non-contiguous) NUMA node IDs
>>> CPU-less NUMA nodes
>>> CPU-less and memory-less NUMA nodes
>>>
>>> For example:
>>>
>>> available: 3 nodes (0,2-3)
>>>
>>> node 0: CPUs present
>>> node 2: CPU-less
>>> node 3: CPU-less
>>>
>>> In this case, the driver creates a CPU topology device only for node 
>>> 0 and
>>> assigns a single CPU proximity domain (0). However, the GPU VCRAT still
>>> references the NUMA node ID to which the GPU is attached (for 
>>> example, node 3)
>>> in the proximity_domain_to field. Since no corresponding CPU 
>>> proximity domain
>>> exists for node 3, the parser cannot find a matching proximity 
>>> domain during
>>> VCRAT parsing, causing topology initialization to fail.
>>
>> Hi Tom, 
>
>
> Hi Felix,
>
>
>>
>> Thank you for the explanation and the patch series. I may have some 
>> gaps in my understanding that I would like to clarify. In my mind, I 
>> was using "proximity domain" and "NUMA node" interchangeably. You're 
>> demonstrating that they are not the same thing. Is that just a 
>> different way of labeling the same thing, or are NUMA nodes and 
>> proximity domains fundamentally different concepts.
>>
>> Your code in patch 5 (kfd_proximity_domain_to_numa_node and 
>> kfd_numa_node_to_proximity_domain) seems to imply that there is, in 
>> fact, a 1:1 mapping, as it assumes that there is a unique translation 
>> in both directions. Am I missing something?
>
>
>
> Thanks for the comment.
>
> IIUC, NUMA node IDs and proximity domains are different numbering 
> schemes. We have proximity domains for both CPU and GPU devices. CPU 
> proximity domains start from 0, and GPU proximity domains start after 
> the last CPU proximity domain.
>
> If the NUMA node IDs are contiguous, the CPU proximity domains happen 
> to match the NUMA node IDs. However, if the NUMA node IDs are 
> discontiguous, the CPU proximity domains and NUMA node IDs no longer 
> have a one-to-one mapping.
>
>>
>> If they are just different numbering systems, do we really need to 
>> keep track of both proximity domains and NUMA nodes in the KFD 
>> topology? Or would it be sufficient to only track NUMA nodes, if 
>> that's what we really care about in the uAPI (KFD sysfs)?
>
>
>
> Thanks for the suggestion. I also think we don't need to keep track of 
> both the proximity domains and the NUMA node IDs in KFD. Do you think 
> the approach below would be reasonable?
>
> Just to make sure I understand correctly, for CPU devices the 
> proximity domain will be the same as the NUMA node ID, and the GPU 
> proximity domains will start after the last CPU proximity domain.
>
> In that case, the CPU proximity domains and NUMA node IDs will always 
> have a one-to-one mapping, and the GPU proximity domains will follow 
> after them.
>
> For example, if a system has three NUMA nodes and two GPUs:
>
> NUMA node IDs:            0   2   3
> CPU proximity domains:    0   2   3
> GPU proximity domains:    4   5
>
> Would it be okay to proceed with this approach?

Yes, this looks good to me.

Regards,
   Felix


>
> I think this approach should also resolve the driver loading issue.
>
>
> Thanks
> Donet Tom
>
>
>>
>> Thanks,
>>   Felix
>>
>>
>>>
>>> The failure is observed as:
>>>
>>> amdgpu: Virtual CRAT table created for GPU
>>> amdgpu: Error parsing VCRAT
>>> kfd: amdgpu: Error adding device to topology
>>> kfd: amdgpu: Error initializing KFD node
>>>
>>>
>>> Since every online NUMA node is a valid topology object and can contain
>>> CPUs, memory, I/O links, or any combination of these, topology devices
>>> and proximity domains should be created for every online NUMA node
>>> rather than only for NUMA nodes that contain CPUs.
>>>
>>> To address this, this series introduces a new VCRAT subtype that
>>> records the mapping between the NUMA node ID and the generated VCRAT
>>> proximity domain. When topology devices are created, this information
>>> is stored in the corresponding topology device, allowing the driver to
>>> translate a NUMA node ID into its associated proximity domain whenever
>>> required.
>>>
>>> Returning to the previous example, the system contains three online
>>> NUMA nodes, so three CPU topology devices and three proximity domains
>>> are created, even though only one NUMA node contains CPUs. The NUMA
>>> node ID is stored in each topology device together with its generated
>>> proximity domain.
>>>
>>> Later, when the GPU VCRAT is generated, the driver only knows the NUMA
>>> node ID to which the GPU is attached (for example, node 3). Instead of
>>> assuming that the NUMA node ID is equal to the proximity domain, the
>>> driver walks the existing topology devices to locate the corresponding
>>> NUMA node and retrieves its generated proximity domain. In this
>>> example, NUMA node 3 maps to proximity domain 2, so
>>> proximity_domain_to is populated with the correct value.
>>>
>>> Since the GPU I/O link now references a valid proximity domain, VCRAT
>>> parsing completes successfully and topology initialization proceeds
>>> without errors on systems with sparse NUMA node IDs, CPU-less NUMA
>>> nodes, and CPU-less/memory-less NUMA nodes.
>>>
>>> This series consists of the following patches:
>>>
>>> Patch 1 removes an unused argument from
>>> kfd_create_crat_image_virtual() as a preparatory cleanup.
>>>
>>> Patch 2 introduces a new VCRAT NUMA affinity subtype that stores the
>>> NUMA node ID and its corresponding proximity domain.
>>>
>>> Patch 3 populates the NUMA affinity entries during VCRAT generation
>>> for every online NUMA node.
>>>
>>> Patch 4 parses the NUMA affinity entries from the VCRAT and stores the
>>> NUMA node ID and proximity domain in the corresponding topology
>>> device.
>>>
>>> Patch 5 fixes GPU VCRAT proximity domain mappings by translating the
>>> GPU's NUMA node ID to the corresponding CPU proximity domain before
>>> programming proximity_domain_to.
>>>
>>> Patch 6 creates proximity domains and VCRAT entries for all online
>>> NUMA nodes, including CPU-less and memory-less nodes.
>>>
>>> Patch 7 adds a numa_node sysfs attribute for CPU and GPU topology
>>> devices, exposing the associated NUMA node through the topology sysfs
>>> interface.
>>>
>>> Please note that the changes in this series are on a best effort 
>>> basis from our
>>> end. Therefore, requesting the amd-gfx community (who have deeper 
>>> knowledge of the
>>> HW & SW stack) to kindly help with the review and provide feedback / 
>>> comments on
>>> these patches
>>>
>>> Donet Tom (7):
>>>    drm/amdgpu: Remove unused argument from 
>>> kfd_create_crat_image_virtual
>>>    drm/amdgpu: Add VCRAT NUMA affinity entry
>>>    drm/amdgpu: Populate NUMA affinity entries in VCRAT
>>>    drm/amdgpu: Parse NUMA affinity entries from VCRAT
>>>    drm/amdgpu: Fix VCRAT proximity domain mappings for GPU nodes
>>>    drm/amdgpu: Create proximity domains and VCRAT entries for CPU-less
>>>      and memory-less NUMA nodes
>>>    drm/amdgpu: Add numa_node in cpu/gpu topology device sysfs entry
>>>
>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.c     | 152 
>>> +++++++++++++++-------
>>>   drivers/gpu/drm/amd/amdkfd/kfd_crat.h     |  19 ++-
>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.c |  49 +++++--
>>>   drivers/gpu/drm/amd/amdkfd/kfd_topology.h |   4 +
>>>   4 files changed, 168 insertions(+), 56 deletions(-)
>>>