Re: Why choosing AF_XDP (busy polling) over DPDK

On 12/14/22 16:31, Magnus Karlsson wrote:
> On Wed, Dec 14, 2022 at 4:16 PM Federico Parola
> <[email protected]> wrote:
>>
>> On 12/14/22 10:00, Magnus Karlsson wrote:
>>> On Wed, Dec 14, 2022 at 9:16 AM Federico Parola
>>> <[email protected]> wrote:
>>>>
>>>> Hello everyone,
>>>> in a context in which all traffic is sent to user space, and the main
>>>> concern is performance (so AF_XDP busy polling is the best candidate),
>>>> is there a reason to choose AF_XDP instead of DPDK for packet I/O, given
>>>> that the latter is still much faster?
>>>
>>> The choice is not AF_XDP vs DPDK, since DPDK can run on top of AF_XDP.
>>> The choice is do you want to use user-space drivers or kernel space
>>> drivers. There are many pros and cons with both approaches. The two
>>> main advantages of user-space drivers in DPDK are that they are faster
>>> than their kernel counterparts and you get access to many NIC features
>>> that are not available right now from user-space if you are using a
>>> kernel driver. Kernel drivers, on the other hand, enables you to
>>> deploy your application on any Linux system (from a certain version)
>>> and does not force you into the strict execution model of DPDK (which
>>> is what most people use with user-space drivers). Everything in Linux
>>> is still usable and it is easy to have a system that mixes data-plane
>>> processing with other types of workloads. If the only thing you care
>>> about is max performance of a pure packet processing workload on a
>>> system you have complete control over, the choice is easy: go with a
>>> user-space driver and DPDK. If this is not your ultimate goal, then
>>> XDP and AF_XDP might be something to consider.
>> Thanks for your answer Magnus.
>>
>> - Concerning deploying an application on any Linux system, what could be
>> the limitations of DPDK userspace drivers that AF_XDP doesn't have? The
>> use of hugepages?
> 
> That would be one. The requirement of shared memory, forced unity
> mmap() calls that might fail, lack of binary forward compatibility,
> etc.
> 
>> - Concerning a system that mixes data-plane processing with other types
>> of workloads, do you refer only to the capability of XDP to steer
>> traffic either towards userspace with AF_XDP or let it proceed to the
>> kernel? Or are there other features of DPDK userspace drivers that would
>> impact this kind of system? Maybe the use of busy polling is too
>> aggressive to share the CPU with other workloads?
> 
> Yes, busy-polling and sharing a core is not a good idea in any system.
> But there is a good solution to this in DPDK that uses capabilities in
> newer CPUs. Basically, if you use DPDK you should adhere to its strict
> model of one process per core, otherwise you will likely get into
> trouble. Either by performance dropping significantly because you
> enable things like interrupts or 4K pages in DPDK, or because you do
> things like trying to run to DPDK polling threads on the same core,
> which might result in deadlock. If I am not mistaken, there is a
> concept in DPDK called service cores in which you can share a core
> between a normal process and a DPDK data plane thread. But it requires
> you to program specifically for it.
> 
> If you want to steer part of the traffic to the kernel, you are better
> off using XDP. Bouncing things through user-space is expensive, DPDK
> or not.
> 
>>>
>>> There are likely some good write-ups on the Internet about this.
>>>
>>> /Magnus
>>>
>>>> Best regards,
>>>> Federico Parola

Thank you very much for this information Magnus.

On 12/14/22 20:46, Andrey Slastenov wrote:> Hello everyone
 >
 > Check out this short article about why XDP was chosen over DPDK. The
 > reasoning for this decision is also explained.
 >
 > 
https://gcore.com/blog/how-we-use-regular-expressions-in-xdp-for-packet-filtering/
 >
 >
 > BR, Andrey

Thanks for sharing this article Andrey.

Best regards,
Federico