Re: AF_XDP not transmitting frames immediately

+Vinicius

On Tue, 2021-12-14 at 10:40 +0000, Karlsson, Magnus wrote:
> Adding Ederson and Maciej.
> 
> > -----Original Message-----
> > From: Jesper Dangaard Brouer <[email protected]>
> > Sent: Tuesday, December 14, 2021 11:32 AM
> > To: Karlsson, Magnus <[email protected]>; Björn Töpel
> > <[email protected]>
> > Cc: Brouer, Jesper <[email protected]>; Xdp <xdp-
> > [email protected]>; Ong, Boon Leong
> > <[email protected]>;
> > Joao Pedro Barros Silva <[email protected]>; Diogo Alexandre Da
> > Silva Lima
> > <[email protected]>
> > Subject: Re: AF_XDP not transmitting frames immediately
> > 
> > 
> > 
> > On 14/12/2021 09.07, Karlsson, Magnus wrote:
> > > 
> > > 
> > > > -----Original Message----- From: Jesper Dangaard Brouer
> > > > <[email protected]> Sent: Monday, December 13, 2021 10:04 PM
> > > > To:
> > > > Karlsson, Magnus <[email protected]>; Björn Töpel
> > > > <[email protected]> Cc: Brouer, Jesper <[email protected]>; Xdp
> > > > <xdp- [email protected]>; Ong, Boon Leong
> > > > <[email protected]>; Joao Pedro Barros Silva
> > > > <[email protected]>; Diogo Alexandre Da Silva Lima
> > > > <[email protected]>
> > > > Subject: AF_XDP not transmitting frames immediately
> > > > 
> > > > Hi Magnus and Bjørn,
> > > > 
> > > > I'm coding on an AF_XDP program[1] that need to send (a bulk of
> > > > packets) in a short time-window (related to Time-Triggered
> > > > Ethernet).
> > > > 
> > > > My observations are that AF_XDP doesn't send the frames
> > > > immediately.
> > > > And yes, I do call sendto() to trigger a TX kick. In zero-copy
> > > > mode
> > > > this is particular bad.  My program want to send 4 packets in a
> > > > burst, but I'm observing 8 packets grouped together on the
> > > > receiving
> > > > host.
> > > > 
> > > > Is the a known property of AF_XDP?
> > > 
> > > Nope! It is supposed to be able to send one packet at a time,
> > > though I
> > > have several times seen bugs in the drivers where the batching
> > > behavior shines through like this, and once a bug in the core
> > > code.
> > > There is even a test these days for just sending a single packet,
> > 
> > Where is that test in the kernel tree?
> 
> In tools/testing/selftests/bpf/xdpxceiver.c. It is the
> RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates on
> veth currently.
> 
> > > since we have had issues with this in the past. That test does
> > > pass in
> > > bpf-next, but it is only run with the veth driver that does not
> > > support zero-copy so could still be an issue. What driver are you
> > > using in zero-copy mode and what kernel version are you on?
> > 
> > Driver: igc with Intel chip i225
> 
> Have never tried this one personally. Do not know if I have one in
> the lab but let me check.
> 
> Ederson, do you have any experience with this card and if so, have
> you seen something similar?

Not sure. I wonder how small is the interval Jesper is using. I imagine
that if it's too small, the interrupt generated to trigger the tx could
end up serving more than one packet.

Vinicius should have more prompt access to i225 - could you please help
on this?
> 
> > Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT
> >    - Devel branch at commit 6d3b1b069946 (v5.15-12802-
> > g6d3b1b069946)
> > 
> > > > How can I get AF_XDP to "flush" TX packets when calling
> > > > sendto()?
> > > > Should we add another flag than the current MSG_DONTWAIT?
> > > 
> > > In zero-copy mode with softirq driver processing (not busy poll),
> > > a
> > > sendto will just trigger the xsk_wakeup ndo that schedules napi
> > > unless
> > > it is already executing. It is up to the driver to then get
> > > packets
> > > from the Tx ring and put them on the HW and make sure they are
> > > sent.
> > > Barring any HW quirks, sending one packets should be perfectly
> > > fine.
> > 
> > I will investigate driver level issues.
> > 
> > I have other (100G) NICs in my testlab, but I'm using these 1G NICs
> > because
> > they support hardware timestamping, which allows me to investigate
> > these
> > timing issues.
> > I'll find a way to see of other drivers behave differently.
> 
> Would be great if you could check if the problem also exists on e.g.
> ice. 
> 
> > > > Hint, I'm using tcpdump hardware timestamping on receiving hist
> > > > via
> > > > cmdline:
> > > > 
> > > > tcpdump -vv -s0 -ni eth1 -j adapter_unsynced
> > > > --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42
> > > > 
> > > > Notice[1] on specific branch: [1]
> > > > https://github.com/xdp-project/bpf-
> > examples/tree/vestas03_AF_XDP_exam
> > > > ple/AF_XDP-interaction
> > > 
> > 
> > Thanks for your feedback,
> > --Jesper
>