Re: AF_XDP sendto kick returning EPERM

On Mon, May 3, 2021 at 1:54 PM Magnus Karlsson
<[email protected]> wrote:
>
> On Thu, Apr 29, 2021 at 5:47 PM Srivats P <[email protected]> wrote:
> >
> > On Tue, Apr 27, 2021 at 12:58 PM Magnus Karlsson
> > <[email protected]> wrote:
> > >
> > > On Fri, Apr 23, 2021 at 5:44 PM Srivats P <[email protected]> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm using sendto() to kick tx in my AF_XDP program after I submit
> > > > descriptors to the tx ring -
> > > >
> > > > ret = sendto(xsk_socket__fd(xsk_), NULL, 0, MSG_DONTWAIT, NULL, 0);
> > > >
> > > > However, I'm receiving EPERM as the return value every time. AFAIK
> > > > this is not an expected return value. Since this is with i40e, I
> > > > checked i40e_xsk_wakeup() - but that also doesn't return EPERM. I am
> > > > running as root and I don't see any problems with creating the xsk,
> > > > configuring umem etc.
> > > >
> > > > Also, no packets seem to go out either.
> > > >
> > > > # uname -a
> > > > Linux Ostinato-1 5.11.15-1-default #1 SMP Fri Apr 16 16:47:34 UTC 2021
> > > > (64fb5bf) x86_64 x86_64 x86_64 GNU/Linux
> > > >
> > > > I don't see the problem on another machine with i40e but older kernel 5.4 series
> > > >
> > > > Any suggestions on what to look for or how to proceed?
> > >
> > > Weird. Have not seen this before. What is your command line for
> > > xdpsock? Is it unmodified?
> >
> > This is not xdpsock, but my own AF_XDP program.
> >
> > >
> > > Using bpftrace, we can get the call stack of xsk_sendmsg. Somewhere in
> > > this stack there must be an EPERM. You can run the same command on
> > > your system, but use ftrace to see what a sendto call hits. Then see
> > > where the code terminates.
> > >
> > > mkarlsso@kurt:~/src/dna-linux$ sudo bpftrace -e 'kprobe:xsk_sendmsg {
> > > @[kstack()] = count(); }'
> > > Attaching 1 probe...
> > > ^C
> > >
> > > @[
> > >     xsk_sendmsg+1
> > >     sock_sendmsg+94
> > >     __sys_sendto+238
> > >     __x64_sys_sendto+37
> > >     do_syscall_64+51
> > >     entry_SYSCALL_64_after_hwframe+68
> > > ]: 2244805
> >
> > Ostinato-1:~ # bpftrace -e 'kprobe:xsk_sendmsg {
> > @[kstack()] = count(); }'
> > Attaching 1 probe...^C@[
> >     xsk_sendmsg+1
> >     sock_sendmsg+94
> >     __sys_sendto+238
> >     __x64_sys_sendto+37
> >     do_syscall_64+51
> >     entry_SYSCALL_64_after_hwframe+68
> > ]: 1253307
> >
> > Which doesn't seem to suggest any error - I've looked at the source
> > code for all these functions, but don't see any reference to EPERM.
>
> It must be in there somewhere :-). Could you plesae use ftrace
> (through perf for example) and trace all functions that a sendto hits
> in your case? Then we might see what it hits.
>
> Are you running in SKB mode or in zero-copy mode? Guess it is
> zero-copy from your mail, but just want to verify. Does Rx work as
> expected?
>
> Could you share your AF_XDP program?

After some experimentation and a lot of head-scratching, I found part
of the problem last night. The sendto() was not returning EPERM (-1),
but ENXIO (-6) - I was mistakenly printing the return value of the
sento() call (which always returns -1 in case of failure), instead of
errno (duh!).

Looking at the code, I see ENXIO is returned if the xsk is unbound.
I'm still investigating this and will post an update soon. The problem
is happening at a customer end and there's some delay and follow up
required to get the logs.

Srivats