Re: AF_XDP not transmitting frames immediately

On 14/12/2021 12.25, Maciej Fijalkowski wrote:
> On Tue, Dec 14, 2021 at 10:40:05AM +0000, Karlsson, Magnus wrote:
>> Adding Ederson and Maciej.
>>
>>> On 14/12/2021 09.07, Karlsson, Magnus wrote:
>>>>
>>>>
>>>>> -----Original Message----- From: Jesper Dangaard Brouer
>>>>> <[email protected]> Sent: Monday, December 13, 2021 10:04 PM To:
>>>>> Karlsson, Magnus <[email protected]>; Björn Töpel
>>>>> <[email protected]> Cc: Brouer, Jesper <[email protected]>; Xdp
>>>>> <xdp- [email protected]>; Ong, Boon Leong
>>>>> <[email protected]>; Joao Pedro Barros Silva
>>>>> <[email protected]>; Diogo Alexandre Da Silva Lima <[email protected]>
>>>>> Subject: AF_XDP not transmitting frames immediately
>>>>>
>>>>> Hi Magnus and Bjørn,
>>>>>
>>>>> I'm coding on an AF_XDP program[1] that need to send (a bulk of
>>>>> packets) in a short time-window (related to Time-Triggered Ethernet).
>>>>>
>>>>> My observations are that AF_XDP doesn't send the frames immediately.
>>>>> And yes, I do call sendto() to trigger a TX kick. In zero-copy mode
>>>>> this is particular bad.  My program want to send 4 packets in a
>>>>> burst, but I'm observing 8 packets grouped together on the receiving
>>>>> host.
>>>>>
>>>>> Is the a known property of AF_XDP?
>>>>
>>>> Nope! It is supposed to be able to send one packet at a time, though I
>>>> have several times seen bugs in the drivers where the batching
>>>> behavior shines through like this, and once a bug in the core code.
>>>> There is even a test these days for just sending a single packet,
>>>
>>> Where is that test in the kernel tree?
>>
>> In tools/testing/selftests/bpf/xdpxceiver.c. It is the RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates on veth currently.
> 
> I'd say it's driver's fault. Magnus fixed something similar for i40e:
> https://lore.kernel.org/netdev/[email protected]/

Thanks for that hint.

> 
> We don't have currently igc HW on our side to dig this :<

I suspected Boon Leong (cc) would have this hardware.

>>
>>>> since we have had issues with this in the past. That test does pass in
>>>> bpf-next, but it is only run with the veth driver that does not
>>>> support zero-copy so could still be an issue. What driver are you
>>>> using in zero-copy mode and what kernel version are you on?
>>>
>>> Driver: igc with Intel chip i225
>>
>> Have never tried this one personally. Do not know if I have one in the lab but let me check.
>>
>> Ederson, do you have any experience with this card and if so, have you seen something similar?
>>
>>> Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT
>>>    - Devel branch at commit 6d3b1b069946 (v5.15-12802-g6d3b1b069946)
>>>
>>>>> How can I get AF_XDP to "flush" TX packets when calling sendto()?
>>>>> Should we add another flag than the current MSG_DONTWAIT?
>>>>
>>>> In zero-copy mode with softirq driver processing (not busy poll), a
>>>> sendto will just trigger the xsk_wakeup ndo that schedules napi unless
>>>> it is already executing. It is up to the driver to then get packets
>>>> from the Tx ring and put them on the HW and make sure they are sent.
>>>> Barring any HW quirks, sending one packets should be perfectly fine.
>>>
>>> I will investigate driver level issues.
>>>
>>> I have other (100G) NICs in my testlab, but I'm using these 1G NICs because
>>> they support hardware timestamping, which allows me to investigate these
>>> timing issues.
>>> I'll find a way to see of other drivers behave differently.
>>
>> Would be great if you could check if the problem also exists on e.g. ice.
>>

Having issues getting my ICE hardware to link up.

I tested that driver i40e works as expected. Thus, this is likely an 
issue with the driver.  I will digg some more.


>>>>> Hint, I'm using tcpdump hardware timestamping on receiving hist via
>>>>> cmdline:
>>>>>
>>>>> tcpdump -vv -s0 -ni eth1 -j adapter_unsynced
>>>>> --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42
>>>>>
>>>>> Notice[1] on specific branch: 

[1] 
https://github.com/xdp-project/bpf-examples/tree/vestas03_AF_XDP_example/AF_XDP-interaction

In [1] I tried to play with SO_PREFER_BUSY_POLL, but it didn't make a 
difference.

[2] https://github.com/xdp-project/bpf-examples/commit/3685d5ea93fced

--Jesper