Re: AF_XDP not transmitting frames immediately

On 14/12/2021 09.07, Karlsson, Magnus wrote:
> 
> 
>> -----Original Message----- From: Jesper Dangaard Brouer
>> <[email protected]> Sent: Monday, December 13, 2021 10:04 PM To:
>> Karlsson, Magnus <[email protected]>; Björn Töpel 
>> <[email protected]> Cc: Brouer, Jesper <[email protected]>; Xdp
>> <xdp- [email protected]>; Ong, Boon Leong
>> <[email protected]>; Joao Pedro Barros Silva
>> <[email protected]>; Diogo Alexandre Da Silva Lima 
>> <[email protected]> Subject: AF_XDP not transmitting frames
>> immediately
>> 
>> Hi Magnus and Bjørn,
>> 
>> I'm coding on an AF_XDP program[1] that need to send (a bulk of
>> packets) in a short time-window (related to Time-Triggered
>> Ethernet).
>> 
>> My observations are that AF_XDP doesn't send the frames
>> immediately. And yes, I do call sendto() to trigger a TX kick. In
>> zero-copy mode this is particular bad.  My program want to send 4
>> packets in a burst, but I'm observing 8 packets grouped together on
>> the receiving host.
>> 
>> Is the a known property of AF_XDP?
> 
> Nope! It is supposed to be able to send one packet at a time, though
> I have several times seen bugs in the drivers where the batching
> behavior shines through like this, and once a bug in the core code.
> There is even a test these days for just sending a single packet,

Where is that test in the kernel tree?

> since we have had issues with this in the past. That test does pass
> in bpf-next, but it is only run with the veth driver that does not
> support zero-copy so could still be an issue. What driver are you
> using in zero-copy mode and what kernel version are you on?

Driver: igc with Intel chip i225

Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT
  - Devel branch at commit 6d3b1b069946 (v5.15-12802-g6d3b1b069946)

>> How can I get AF_XDP to "flush" TX packets when calling sendto()? 
>> Should we add another flag than the current MSG_DONTWAIT?
> 
> In zero-copy mode with softirq driver processing (not busy poll), a
> sendto will just trigger the xsk_wakeup ndo that schedules napi
> unless it is already executing. It is up to the driver to then get
> packets from the Tx ring and put them on the HW and make sure they
> are sent. Barring any HW quirks, sending one packets should be
> perfectly fine.

I will investigate driver level issues.

I have other (100G) NICs in my testlab, but I'm using these 1G NICs 
because they support hardware timestamping, which allows me to 
investigate these timing issues.
I'll find a way to see of other drivers behave differently.

>> Hint, I'm using tcpdump hardware timestamping on receiving hist via
>> cmdline:
>> 
>> tcpdump -vv -s0 -ni eth1 -j adapter_unsynced --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42
>> 
>> Notice[1] on specific branch: [1] 
>> https://github.com/xdp-project/bpf-examples/tree/vestas03_AF_XDP_example/AF_XDP-interaction
> 

Thanks for your feedback,
--Jesper