CPU utili incr on kernel migration 4.19 -> 5.4

Hello!

Recently, we tried to migrate from the kernel 4.19.125 to 5.4.17.
During stress tests we observe that on the same hardware (we are using
virtual machines in our on-prem cloud on VMWare) and using the same
XDP application, on the 4.19.125 softirq CPU utilization is 23% vs.
the 35% on 5.4.17. Please, note, we are using the FIPS on 5.4.17
kernel.

At the first glance, using linux perf I see some difference, but can't
understand them:

5.4.17 perf top -a

   11.90%  [kernel]                                [k] lookup_nulls_elem_raw
    8.79%  [kernel]                                [k] __do_softirq
    4.97%  [kernel]                                [k]
_raw_spin_unlock_irqrestore
    3.62%  [kernel]                                [k] memcpy_orig
    3.39%  bpf_prog_2e039a5b937c8960               [k] bpf_prog_2e039a5b937c8960
    2.66%  [kernel]                                [k] native_write_msr
    2.34%  [kernel]                                [k] vmxnet3_rq_rx_complete
    1.87%  [kernel]                                [k] arch_local_irq_restore
    1.71%  [kernel]                                [k] __htab_map_lookup_elem
    1.33%  [kernel]                                [k] page_frag_free
    1.31%  [kernel]                                [k] kmem_cache_free
    1.28%  [kernel]                                [k] vmxnet3_poll_rx_only
    1.13%  [kernel]                                [k] finish_task_switch

4.19.125 perf top -a

   12.90%  [kernel]                                [k] lookup_nulls_elem_raw
    9.11%  [kernel]                                [k] __softirqentry_text_start
    4.36%  [kernel]                                [k]
_raw_spin_unlock_irqrestore
    3.66%  [kernel]                                [k] memcpy_orig
    3.03%  [kernel]                                [k] native_write_msr
    2.69%  [kernel]                                [k] vmxnet3_rq_rx_complete
    2.34%  [kernel]                                [k] _lock+0x800000005cc2
    1.91%  [kernel]                                [k] arch_local_irq_restore
    1.53%  [kernel]                                [k] __htab_map_lookup_elem
    1.36%  [kernel]                                [k] __slab_free
    1.31%  [kernel]                                [k] vmxnet3_poll_rx_only
    1.24%  [kernel]                                [k] page_frag_free
    1.14%  [kernel]                                [k] finish_task_switch

Could you help me please to identify softirq CPU utilization increase?

-- 
Alexander Petrovsky