Re: [RESEND PATCH v2 0/4] nvme-tcp: NIC topology aware I/O queue scaling and queue info export

On 7/31/26 10:09 PM, Stanislav Fomichev wrote:
> On 07/31, Nilay Shroff wrote:
>> Hi,
>>
>> This is a resend of the previous series to include the networking maintainers
>> and mailing list. There are no code or commit message changes since the
>> previous posting.
>>
>> This series has been updated based on the feedback received during LSFMM.
>> The changelog is updated accordingly.
>>
>> The NVMe/TCP host driver currently provisions I/O queues primarily based
>> on CPU availability rather than the capabilities and topology of the
>> underlying network interface.
>>
>> On modern systems with many CPUs but fewer NIC hardware queues, this can
>> lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue,
>> resulting in increased lock contention, cacheline bouncing, and degraded
>> throughput.
>>
>> This RFC proposes a set of changes to better align NVMe/TCP I/O queues
>> with NIC queue resources, and to expose queue/flow information to enable
>> more effective system-level tuning.
>>
>> Key ideas
>> ---------
>>
>> 1. Scale NVMe/TCP I/O queues based on NIC queue count
>>     Instead of relying solely on CPU count, limit the number of I/O workers
>>     to:
>>         min(num_online_cpus, netdev->real_num_{tx,rx}_queues)
>>
>> 2. Improve CPU locality
>>     Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity
>>     to reduce cross-CPU traffic and improve cache locality.
>>
>> 3. Expose queue and flow information via debugfs
>>     Export per-I/O queue information including:
>>         - queue id (qid)
>>         - CPU affinity
>>         - TCP flow (src/dst IP and ports)
> 
> [..]
> 
>>     This enables userspace tools to configure:
>>         - IRQ affinity
>>         - RPS/XPS
>>         - ntuple steering
>>         - or any other scaling as deemed feasible
> 
> Can you expand on this a bit? What specifically helped the most for your
> tuned case?

For my network topology and workload, configuring ntuple steering together with
IRQ affinity and XPS provided the most benefit. The NIC used for testing supports
ntuple filters, which allowed me to steer the TCP flow associated with an NVMe/TCP
I/O queue to a specific NIC RX queue. I then configured the IRQ affinity for that
RX queue and XPS for the corresponding TX path so that the NVMe/TCP I/O queue
processing, RX interrupt processing, and TX/RX packet processing were aligned to
the same CPU as much as possible. This reduced cross-CPU processing for an I/O
flow and provided the best improvement among the tuning combinations I tested.

Thanks,
--Nilay