Re: Question about xdp: how to figure out the throughput is limited by pcie

Hi Andi and Acme,

Regarding below discussion and subj (top-posting as you don't need to
read discussion to answer my perf questions).

Can we somehow use perf to profile things happening in PCIe ?
E.g. Are there any PMU counters "uncore" events for PCIe ?

   Hint, we can list more PMU counter via Andi's ocperf tool[42].
   # sudo ./ocperf list

Could we use the TopDown [toplev] model, to indicate/detect that the
PCIe device (or PCIe root complex) is the bottleneck?

   Hint, try out the [toplev] tool looking at specific core under-load
   # sudo ./toplev.py -I 3000 -l3 -a --show-sample --core C2

--Jesper

  [toplev] https://github.com/andikleen/pmu-tools/wiki/toplev-manual
  [42] https://github.com/andikleen/pmu-tools

On 13/04/2023 04.54, Qiongwen Xu wrote:
> Hi Jesper,
> 
> Thanks for the detailed reply and sharing these helpful materials/papers with us!
> 
> After enabling rx_cqe_compress, the throughput in our experiment increases from
> 70+Mpps to 85 Mpps. We also tried to use the counter "rx_discards_phy". The counter
> increases in both cpu-limited and pcie-limited experiments, i.e., in the experiment
> which is only cpu-limited can also increase the counter. We are looking for any
> counter that can separate cpu- and pcie-limited cases. Regarding the [pcie-bench] tool,
> unfortunately, we are not able to use it, as it requires fpga hardware.
> 
> Thanks,
> Qiongwen
> 
> From: Jesper Dangaard Brouer <[email protected]>
> Date: Sunday, April 9, 2023 at 11:46 AM
> Subject: Re: Question about xdp: how to figure out the throughput is limited by pcie
> (answered inline below)
> 
> On 07/04/2023 03.46, Qiongwen Xu wrote:
>> Dear XDP experts,
>>
>> I am a PhD student at Rutgers. Recently, I have been reading the XDP
>> paper "The eXpress Data Path: Fast Programmable Packet Processing
>> in the Operating System Kernel". In section 4.1 and 4.3, you mention
>> the throughputs of xdp programs (packet drop and packet forwarding)
>> are limited by the PCIe (e.g., "Both scale their performance linearly
>> until they approach the global performance limit of the PCI bus").
> 
> Most of the article[1][2] authors are likely this mailing list,
> including me. (Sad to see we called it "PCI *bus*" and not just PCIe).
> 
>> I am curious about how you figured out it was the PCIe limitation.
> 
> It is worth noting that the PCIe limitation shown in article is related
> to number of PCIe transactions with small packets (Ethernet minimum
> frame size 64 Bytes). (Thus meaning NOT bandwidth related).
> 
> The observations that lead to the PCIe limitation conclusion:
> A single CPU doing XDP_DROP (25Mpps) was using 100% CPU time (runtime
> attributed to ksoftirqd).  When we scaled up XDP_DROP to run on more
> CPUs we saw something strange[3].  It scaled linear to 3 CPUs, and at 4
> CPUs each CPU started to process less packets per sec (pps) and total
> (86Mpps) stayed the same.  Even more strange the CPUs wasn't using 100%
> CPU any-longer, CPUs had "time" to idle.  Looking at ethtool stats, we
> noticed the counter "rx_discards_phy", which (we were told) happens when
> PCIe causes backpressure.
> 
> What confirmed the PCIe (transactions) bottleneck was[4] when we
> discovered enabling the mlx5 priv-flags rx_cqe_compress=on (and
> rx_striding_rq=off) changed the total limit (86Mpps to 108Mpps),
> as rx_cqe_compress reduce the transactions on PCIe by compressing the RX
> descriptors.  Thus, confirming this was related to PCIe.
> 
> 
>   > Is there any tool or method to check this?
> 
> I *highly* recommend that you read this article [pci1][pci2]:
>    - Title: "Understanding PCIe performance for end host networking"
> 
> I wish we had read and referenced this article in ours (but both
> happened in 2018).  They give a theoretical model for PCIe, both
> bandwidth and latency.  That could be used to explain our PCIe
> observations. They also released their [pcie-bench] tool.
> 
> I wish more (kernel) performance people understood, that PCIe is a
> protocol (3-layers: physical, data link layer (DLL) and Transaction
> Layer Packets (TLP)), that is used between the device and host
> OS-driver.  In networking usually ignores this PCIe protocol step, with
> associated protocol overheads, which actually causes a network packet to
> be split into smaller PCIe TLP "packets" with their own PCIe level
> headers. Besides the packet data itself, the PCIe protocol is used for
> reading TX desc (seen from device) and writing RX desc (seen from
> device), and read/update queue pointers.
> 
> It might surprise people that article [pci1] shows, that PCIe (128B
> payload) introduces a latency around 600ns (nanosec), which is
> significantly larger than the inter-packet gap needed for wirespeed
> networking.  Thus, latency hiding happens "behind our back", via the
> device and DMA engine have to keep many transactions in-flight to
> utilize the NIC (yet another hidden queue in the system).
> 
> --Jesper
> 
> Links:
> 
>    [1] https://dl.acm.org/doi/10.1145/3281411.3281443
>    [2] https://github.com/xdp-project/xdp-paper
>    [3]
> https://github.com/xdp-project/xdp-paper/blob/master/benchmarks/bench02_xdp_drop.org
>    [4]
> https://github.com/xdp-project/xdp-paper/blob/master/benchmarks/bench01_baseline.org
> 
> Read this article:
>    [pci0] https://dl.acm.org/doi/10.1145/3230543.3230560
>    [pci1]
> https://www.cl.cam.ac.uk/research/srg/netos/projects/pcie-bench/neugebauer2018understanding.pdf
>    [pci2] https://www.cl.cam.ac.uk/research/srg/netos/projects/pcie-bench/
>    [pcie-bench] https://github.com/pcie-bench/pcie-model
>