Re: RFC: using NIC DMA ring as tracing buffer?

Hi,

On Wed, Apr 17, 2024 at 6:51=E2=80=AFPM Alexander Aring <[email protected]=
m> wrote:
>
> Hi Mathieu,
>
> On Tue, Apr 16, 2024 at 10:49=E2=80=AFAM Mathieu Desnoyers
> <[email protected]> wrote:
> >
> > On 2024-04-16 10:41, Alexander Aring wrote:
> > > Hi all,
> > >
> > > I have an idea about using the DMA rings of a networking card as a
> > > tracing buffer.
> >
> > The first thing I'd be worried about is overhead caused by
> > false-sharing if you trace a large SMP/many-core system all
> > into the same global NIC buffer. The cache lines will bounce
> > left and right which is bad performance-wise.
> >
>
> modern NICs have several tx and rx dma rings. There is an ethtool
> command to query the number:
>
> sudo ethtool -g $IFACE
>
> so it depends on the hardware if you can pin one or more rings on a
> per-cpu buffer basis.
>
> I hope that somehow helps to make this idea less worrisome.
>

What I can think about that makes it worrisome is NUMA. In case of an
E-PCI connected NIC, the NICs buffer associated with a core needs to
be in the same NUMA node. You need to set up your system right, even
with 2 nics or more, to make it work "better".

- Alex