i40e: low tx rates for larger packet sizes

Hi,

Trying AF_XDP xdpsock txonly on an i40e (XL710), I'm seeing some
behaviour across packet size that I need help to understand.

All tests with zero-copy mode (similar results with busy poll mode
also). The i40e driver version is the native kernel version 6.8.0-47
included in Ubuntu 24.04. Flow control tx/rx is off.

PktSize  Max-pps  Actual-pps
                                                                    64
   59.5M      ~18.9M
                                                                512
 9.4M      ~7.5M
                                                             1024
4.8M      ~4.6M
                                                            1518
3.3M      ~3.0M

Output logs below

Since 64 byte packets can do around ~18.9M, I was expecting larger
packets (see 512 and 1518 above) to do line rate i.e. do close to
max-pps but they do less than that.

Is this expected?

What factors could be playing a role in a lower rate for larger packet size?

To determine how many cores/queues I need if I need to transmit X Gbps
- what factors do I need to use?

Srivats

$ sudo ./xdpsock -t -i enp1s0f0np0 -z -d5 -s64

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 1,88,28,546    1,88,39,104

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 1,88,25,658    3,76,73,792

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 1,92,29,105    5,69,07,520

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 1,96,44,299    7,65,55,904

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 1,96,46,711    9,62,06,336

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           0.00
rx                 0              0
tx                 0              9,62,06,720


$ sudo ./xdpsock -t -i enp1s0f0np0 -z -d5 -s512

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 75,09,247      75,11,104

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 74,89,511      1,50,01,856

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 74,98,753      2,25,01,568

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 74,86,126      2,99,88,672

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 74,86,807      3,74,76,416

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           0.00
rx                 0              0
tx                 0              3,74,76,416


$ sudo ./xdpsock -t -i enp1s0f0np0 -z -d5 -s1024

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 46,01,574      46,02,496

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 45,95,655      91,98,912

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 45,95,058      1,37,94,432

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 45,97,400      1,83,92,256

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 45,84,541      2,29,77,152

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           0.00
rx                 0              0
tx                 0              2,29,77,216

$ sudo ./xdpsock -t -i enp1s0f0np0 -z -d5 -s1518

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 30,00,232      30,00,960

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 29,94,793      59,96,224

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 29,95,064      89,91,616

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 29,96,147      1,19,88,160

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           1.00
rx                 0              0
tx                 29,94,991      1,49,83,552

 sock0@enp1s0f0np0:0 txonly xdp-drv
                   pps            pkts           0.00
rx                 0              0
tx                 0              1,49,83,552