Re: AF_XDP not transmitting frames immediately

On 15/12/2021 02.08, Desouza, Ederson wrote:
> +Vinicius
> 
> On Tue, 2021-12-14 at 10:40 +0000, Karlsson, Magnus wrote:
>> Adding Ederson and Maciej.
>> 
>>> On 14/12/2021 09.07, Karlsson, Magnus wrote:
>>>> 
>>>> 
>>>>> From: Jesper Dangaard Brouer 
>>>>> 
>>>>> Hi Magnus and Bjørn,
>>>>> 
>>>>> I'm coding on an AF_XDP program[1] that need to send (a bulk
>>>>> of packets) in a short time-window (related to
>>>>> Time-Triggered Ethernet).
>>>>> 
>>>>> My observations are that AF_XDP doesn't send the frames 
>>>>> immediately. And yes, I do call sendto() to trigger a TX
>>>>> kick. In zero-copy mode this is particular bad.  My program
>>>>> want to send 4 packets in a burst, but I'm observing 8
>>>>> packets grouped together on the receiving host.
>>>>> 
>>>>> Is the a known property of AF_XDP?
>>>> 
>>>> Nope! It is supposed to be able to send one packet at a time, 
>>>> though I have several times seen bugs in the drivers where the
>>>> batching behavior shines through like this, and once a bug in
>>>> the core code. There is even a test these days for just sending
>>>> a single packet,
>>> 
>>> Where is that test in the kernel tree?
>> 
>> In tools/testing/selftests/bpf/xdpxceiver.c. It is the 
>> RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates
>> on veth currently.
>> 
>>>> since we have had issues with this in the past. That test does 
>>>> pass in bpf-next, but it is only run with the veth driver that
>>>> does not support zero-copy so could still be an issue. What
>>>> driver are you using in zero-copy mode and what kernel version
>>>> are you on?
>>> 
>>> Driver: igc with Intel chip i225
>> 
>> Have never tried this one personally. Do not know if I have one in 
>> the lab but let me check.
>> 
>> Ederson, do you have any experience with this card and if so, have 
>> you seen something similar?
> 
> Not sure. I wonder how small is the interval Jesper is using. I
> imagine that if it's too small, the interrupt generated to trigger
> the tx could end up serving more than one packet.

The interval is currently 1 second (code here[0]), thus the TX interrupt 
should have plenty of time to trigger.

[0] 
https://github.com/xdp-project/bpf-examples/blob/vestas03_AF_XDP_example/AF_XDP-interaction/af_xdp_user.c#L1129


> Vinicius should have more prompt access to i225 - could you please
> help on this?

My reproducer[1] need option --zero-copy to enable the error case, as it 
defaults to 'copy-mode'.
It is only in zero-copy mode for igc/i225 that I see the behavior of 8 
packet bulking, when expecting/sending bulks of 4 packets ever second.


[1] 
https://github.com/xdp-project/bpf-example/tree/vestas03_AF_XDP_example/AF_XDP-interaction

--Jesper