Re: AF_XDP busy poll receives packets in batches of 8 on i40e

On 18/03/22 00:26, Jay Vosburgh wrote:
> Jesse Brandeburg <[email protected]> wrote:
> 
>> On 3/17/2022 8:34 AM, Federico Parola wrote:
>>>>      We observed similar "batching" behavior on i40e devices late
>>>> last year in ordinary use (not XDP, but using SR-IOV VFs).  We
>>>> instrumented the drivers at the send and receive sides, and determined
>>>> that it appeared to be a behavior of the receiving device itself, i.e.,
>>>> packets 1 - 7 would be held indefinitely (as I recall, no interrupt or
>>>> update of the RX ring pointers) until packet 8 arrives, at which point
>>>> all 8 were delivered simultaneously.
>>>>
>>>>      The issue was evidently in the firmware, and was resolved after
>>>> a firmware upgrade.
>>> Hi Jay,
>>> I just updated the firmware to the latest version (v8.50 from v8.30) but
>>> unfortunately the problem is still there.
>>> However I'm experiencing the problem only when using AF_XDP in busy poll
>>> mode, all other modes (standard AF_XDP and normal packet reception) work
>>> just fine.
>>> Maybe the two problems are correlated in some way.
>>
>> This sounds related to the WB_ON_ITR feature in our hardware. If
>> interrupts are disabled the driver needs to set that bit (and an ITR
>> value) so that packets get written back in a timely manner and don't just
>> wait for a cache line edge (I bet your Cache Line Size value in PCIe space
>> (lspci) is set to 128?)
> 
> 	FWIW, in the case we had, driver changes (5.14 upstream, the
> then-current Intel out of tree driver) didn't change the behavior.  The
> PCI Cache Line Size was 64 bytes; the device was
> 
> 13:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
>          Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 562SFP+ Adapter
> 
> 	From an lspci -vvv perspective, failing and non-failing cards
> differed only in the serial numbers.
> 

I tested it on 3 different XL710 NICs, all updated to the latest 8.50 
firmware, and they all present the same 8 packets "batching" behavior.
How do I check the PCI Cache Line Size (I didn't find such entry in 
lspci -vvv)? Normal cache line size is 64B on my system, but I guess 
they are two different things.

>> Maybe something to chase there? We may also be able to look into it if it
>> wasn't fixed already upstream.
> 
> 	-J
> 
> ---
> 	-Jay Vosburgh, [email protected]