Multi-core scalability problems
Federico Parola <[email protected]>
| Newsgroups | org.kernel.vger.xdp-newbies |
|---|---|
| Message-ID | <VI1PR04MB3104C89EF8DCB98F5330F36C9E040@VI1PR04MB3104.eurprd04.prod.outlook.com> |
Hello, I'm testing the performance of XDP when dropping packets using multiple cores and I'm getting unexpected results. My machine is equipped with a dual port Intel XL710 40 GbE and an Intel Xeon Gold 5120 CPU @ 2.20GHz with 14 cores (HyperThreading disabled), running Ubuntu server 18.04 with kernel 5.8.12. I'm using the xdp_rxq_info program from the kernel tree samples to drop packets. I generate 64 bytes UDP packets with MoonGen for a total of 42 Mpps. Packets are uniformly distributed in different flows (different src port) and I use flow direction rules on the rx NIC to send these flows to different queues/cores. Here are my results: 1 FLOW: Running XDP on dev:enp101s0f0 (ifindex:3) action:XDP_DROP options:no_touch XDP stats CPU pps issue-pps XDP-RX CPU 0 17784270 0 XDP-RX CPU total 17784270 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 17784270 0 rx_queue_index 0:sum 17784270 --- 2 FLOWS: Running XDP on dev:enp101s0f0 (ifindex:3) action:XDP_DROP options:no_touch XDP stats CPU pps issue-pps XDP-RX CPU 0 7016363 0 XDP-RX CPU 1 7017291 0 XDP-RX CPU total 14033655 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 7016366 0 rx_queue_index 0:sum 7016366 rx_queue_index 1:1 7017294 0 rx_queue_index 1:sum 7017294 --- 4 FLOWS: Running XDP on dev:enp101s0f0 (ifindex:3) action:XDP_DROP options:no_touch XDP stats CPU pps issue-pps XDP-RX CPU 0 2359478 0 XDP-RX CPU 1 2358508 0 XDP-RX CPU 2 2357042 0 XDP-RX CPU 3 2355396 0 XDP-RX CPU total 9430425 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 2359474 0 rx_queue_index 0:sum 2359474 rx_queue_index 1:1 2358504 0 rx_queue_index 1:sum 2358504 rx_queue_index 2:2 2357040 0 rx_queue_index 2:sum 2357040 rx_queue_index 3:3 2355392 0 rx_queue_index 3:sum 2355392 I don't understand why overall performance is reducing with the number of cores, according to [1] I would expect it to increase until reaching a maximum value. Is there any parameter I should tune to overcome the problem? Thanks in advance for your help. Federico Parola [1] https://github.com/tohojo/xdp-paper/blob/master/benchmarks/bench02_xdp_drop.org