Re: RSS causing bad forwarding performance?

I think I've found the culprit.

When using RSS the mlx5 driver binds queue interrupts to CPUs. To
simulate what it would do, but without RSS, I've configured both NICs to
have only one queue and then bound the RX NIC to CPU 1 and the TX NIC to
CPU 2. I got 3.3Mpps forwarded this way. Then I've bound the TX NIC to
CPU 1. The forwarding rate immediately dropped to 0.5Mpps.

Both NIC queues on CPU 1:
RX IRQ 546/s
TX IRQ 546/s
dev.mce.1.txstat0tc0.enobuf grows at 3.9M/s
Forwarding at 0.5Mp/s

mce0 queue 0 on CPU 1, mce1 queue 0 on CPU 2:
RX IRQ:   403/s
TX IRQ: 85000/s
dev.mce.1.txstat0tc0.enobuf does not grow
Forwarding at 3.3Mp/s

Let's play with the TX NICs sysctls

dev.mce.1.conf.tx_coalesce_mode=1
dev.mce.1.conf.tx_completion_fact=32
dev.mce.1.conf.tx_coalesce_pkts=128
dev.mce.1.conf.tx_coalesce_usecs=128
TX IRQ:  4300/s
dev.mce.1.txstat0tc0.enobuf grows at 1.6M/s
Forwarding rate 2.19Mp/s

dev.mce.1.conf.tx_queue_size=16384
dev.mce.1.conf.tx_coalesce_mode=0
dev.mce.1.conf.tx_completion_fact=512
dev.mce.1.conf.tx_coalesce_pkts=128
dev.mce.1.conf.tx_coalesce_usecs=128
TX IRQ:  7700/s
dev.mce.1.txstat0tc0.enobuf does not grow
Forwarding at 3.39Mp/s

Now the TX IRQ can be set back to CPU 1 and forwarding rate is at 3.27Mp/s.

Lesson learned: don't forget about interrupt coalescing on the TX side too.

-- 
| pozdrawiam / regards | Powered by Debian and FreeBSD  |
| Kajetan Staszkiewicz |   www: http://tuxpowered.net   |
|                      | matrix: @vegeta:tuxpowered.net |
`----------------------^--------------------------------'

-----BEGIN PGP SIGNATURE-----

wsF5BAABCAAjFiEEfSEIcxY9kDeZvSZsgKrXf6bzAFMFAmk3RqgFAwAAAAAACgkQgKrXf6bzAFOx
9w/9EyJzbSWJDES0UFYUxkxlZZ9k0PixkbM7AVcHgrN3idBXjxrB45lpiTpWwbDDi+wzR08qycC7
Tb/KrZ5DaaDuu1t9PN/RWihPSVUTtseYtSng9MRxr0rZieB/t+gfNQcXnR7wjHEXkZC84v1+oVFB
Atr44upFxk0618GVd+JH+ReZx49TUb5nmYJoCWGaJF7+R9mjLJeGsCdRavY0OZqZqXY4jhDHser4
dytmJS1651KKHNgjqEVtcvQOOstNjrGCCkN2lO5Ts35aNrfnqYwQhiMWpjkY1+D4mTO+lum2bufp
jmFlGr0Ycv2aeJvgFGaeMARmo2ryR43MSXRec5bbTKdVcN6BWzmGB3oHvzvpPj0fab/Ok8jJS/w6
UcyxV2Yspqx1JDRGC8Lb1aTjivN6UhWLpin+Gaep1NcYMvimagL0+2LVHcc7DrRhYUeI28BRcUYb
ZjE1V2MyeiNY0MIPj+w/BjZcszWK1t+4YLMz1w0MdFnbbxbNbzdZpCTgsB2cT0Hx5W+ml+m758ng
fxEbDf4RF3k8I0meGXhQAvpTsAAdpvmJ3pysAzMm1LO6U3L0E4KJ3J0X3WeFZhRRGrNEEw+kf0X/
eZBWjOs/QSWY2C+5kF7BZnoy15AcyWTXw8tk2sgQwCwRA4ntM1o+golI8NW8ExmvKHv/GwaSlaw7
f+I=
=TCfb
-----END PGP SIGNATURE-----