Re: AF_XDP sockets across multiple NIC queues

Great, thanks for the info! I will look into implementing this.

For the time being, I implemented a version of my design with N^2
sockets. I observed that when all traffic is directed to a single NIC
queue, the throughput is higher than when I use all N NIC queues. I am
using spinlocks to guard concurrent access to UMEM and the
fill/completion rings. When I use a single NIC queue, I achieve
~1Mpps; when I use multiple ~550Kpps. Are these numbers reasonable,
and this bad scaling behavior expected?


On Thu, 25 Mar 2021 at 00:24, Magnus Karlsson <[email protected]> wrote:
>
> On Thu, Mar 25, 2021 at 7:25 AM Konstantinos Kaffes <[email protected]> wrote:
> >
> > Hello everyone,
> >
> > I want to write a multi-threaded AF_XDP server where all N threads can
> > read from all N NIC queues. In my design, each thread creates N AF_XDP
> > sockets, each associated with a different queue. I have the following
> > questions:
> >
> > 1. Do sockets associated with the same queue need to share their UMEM
> > area and fill and completion rings?
>
> Yes. In zero-copy mode this is natural since the NIC HW will DMA the
> packet into a umem that was decided long before the packet was even
> received. And this is of course before we even get to pick what socket
> it should go to. This restriction is currently carried over to
> copy-mode, however, in theory there is nothing preventing supporting
> multiple umems on the same netdev and queue id in copy-mode. It is
> just that nobody has implemented support for it.
>
> > 2. Will there be a single XSKMAP holding all N^2 sockets? If yes, what
> > happens if my XDP program redirects a packet to a socket that is
> > associated with a different NIC queue than the one in which the packet
> > arrived?
>
> You can have multiple XSKMAPs but you would in any case have to have
> N^2 sockets in total to be able to cover all cases. Sockets are tied
> to a specific netdev and queue id. If you try to redirect to socket
> with a queue id or netdev that the packet was not received on, it will
> be dropped. Again, for copy-mode, it would from a theoretical
> perspective be perfectly fine to redirect to another queue id and/or
> netdev since the packet is copied anyway. Maybe you want to add
> support for it :-).
>
> > I must mention that I am using the XDP skb mode with copies.
> >
> > Thank you in advance,
> > Kostis



-- 
Kostis Kaffes
PhD Student in Electrical Engineering
Stanford University