Re: lib/group_cpus: make group CPU cluster aware

On 30/07/26 06:58AM, Guo, Wangyang wrote:
>> For higher number of jobs(>=64), we are seeing ~8% regression with this patch.
>>
>> HW info:
>> Intel(R) Core(TM) Ultra 7 265K
>> CPU=20
>> IRQ-CPU are 1:1 mapped.
>
>If IRQ and CPUs are 1:1 mapping (20 IRQs/20 CPUs), the patch would not change anything. Every group is exactly 1 CPU, same as before.

We see its not the case, before this patch, irqs smp_allowed_list was
sequential, but after the patch this changes.

Hardware:
Intel(R) Core(TM) Ultra 7 265K
- 8 P-cores (each its own cluster, own L2)
- 12 E-cores (3 clusters of 4 CPUs sharing L2)
- 1 NUMA node, 20 CPUs total, 21 NVMe queues(admin+io)

Cluster topology (from cluster_cpus_list):
{0} {1} {2} {3} {4} {5} {6} {7} {8-11} {12-15} {16-19}

Before:
/proc/irq/168/smp_affinity_list:0
/proc/irq/169/smp_affinity_list:1
/proc/irq/170/smp_affinity_list:2
/proc/irq/171/smp_affinity_list:3
/proc/irq/172/smp_affinity_list:4
/proc/irq/173/smp_affinity_list:5
/proc/irq/174/smp_affinity_list:6
/proc/irq/175/smp_affinity_list:7
/proc/irq/176/smp_affinity_list:8
/proc/irq/177/smp_affinity_list:9
/proc/irq/178/smp_affinity_list:10
/proc/irq/179/smp_affinity_list:11
/proc/irq/180/smp_affinity_list:12
/proc/irq/181/smp_affinity_list:13
/proc/irq/182/smp_affinity_list:14
/proc/irq/183/smp_affinity_list:15
/proc/irq/184/smp_affinity_list:16
/proc/irq/185/smp_affinity_list:17
/proc/irq/186/smp_affinity_list:18
/proc/irq/187/smp_affinity_list:19

After patch:
/proc/irq/168/smp_affinity_list:1
/proc/irq/169/smp_affinity_list:4
/proc/irq/170/smp_affinity_list:0
/proc/irq/171/smp_affinity_list:5
/proc/irq/172/smp_affinity_list:6
/proc/irq/173/smp_affinity_list:7
/proc/irq/174/smp_affinity_list:3
/proc/irq/175/smp_affinity_list:2
/proc/irq/176/smp_affinity_list:8
/proc/irq/177/smp_affinity_list:9
/proc/irq/178/smp_affinity_list:10
/proc/irq/179/smp_affinity_list:11
/proc/irq/180/smp_affinity_list:12
/proc/irq/181/smp_affinity_list:13
/proc/irq/182/smp_affinity_list:14
/proc/irq/183/smp_affinity_list:15
/proc/irq/184/smp_affinity_list:16
/proc/irq/185/smp_affinity_list:17
/proc/irq/186/smp_affinity_list:18
/proc/irq/187/smp_affinity_list:19

I think this has to do with cache locality effects, when tasks run on
"wrong" CPUs. If you want us to check perf data, we can run that.

Below patch[1] fixes the issue for us. Basically keep the cpu order if
resources are equal, meanwhile keeping the benefit of your patches for
clustering.

[1]
--- a/lib/group_cpus.c
+++ b/lib/group_cpus.c
@@ -109,7 +109,11 @@ static int ncpus_cmp_func(const void *l, const void *r)
{
	const struct node_groups *ln = l;
	const struct node_groups *rn = r;

-       return ln->ncpus - rn->ncpus;
+       if (ln->ncpus != rn->ncpus)
+               return ln->ncpus - rn->ncpus;
+
+       /* Tie-break by id so equal-ncpus clusters keep discovery order */
+       return ln->id - rn->id;
}

Regards,
-Nitesh