[SPDK] Re: 128k sequential read / write (fio) performance with spdk+vpp is not as good as the one with "kernel TCP"

When I tuned vpp for our NIC's I found that the default vpp
configuration wasn't sufficient. Doesn't look like you're controlling
the worker threads. When vpp uses dpdk unless you control which cores
the workers use they'll compete with spdk cores and I found letting
them compete doesn't work well. Instead, assign vpp to dedicated cores
and control the spdk coremask so that it doesn't use the vpp cores. I
matched the vpp worker threads with a number of RSS queues from the
NIC. Other hindrances to performance were multi-seg and tx checksum
offload. You want to use tx cksum offload because the s/w overhead is
very high. This will depend on your h/w but are things to look at.
With this config I was able to get ~800k iops for 4k reads (to nvme
not null). Can't remember the numbers for other sized reads/writes. It
wasn't great but the kernel was worse.

cpu {
  ## Set logical CPU core where main thread runs, if main core is not set
  ## VPP will use core 1 if available
  main-core 0

  ## Set logical CPU core(s) where worker threads are running
  # corelist-workers 2-3,18-19
  corelist-workers 1-3

  ## Specify a number of workers to be created
  ## Workers are pinned to N consecutive CPU cores while skipping
"skip-cores" CPU core(s)
  ## and main thread's CPU core
  workers 3

}

dpdk {
  dev default {
    ## Number of receive queues, enables RSS
    ## Default is 1
    num-rx-queues 3
...
}
  ## Disable multi-segment buffers, improves performance but
  ## disables Jumbo MTU support
  no-multi-seg

  ## Disables UDP / TCP TX checksum offload. Typically needed for use
  ## faster vector PMDs (together with no-multi-seg)
  #no-tx-checksum-offload

}


On Thu, Feb 13, 2020 at 7:57 PM <sejun.kwon(a)samsung.com> wrote:
>
> Hello,
> I'm working on SPDK library + VPP, because some report said that VPP reduces the overhead of network.
> When I test with VPP (with mlx5 poll mode driver) and null device with spdk, 4k performance with VPP is much better than the default(kTCP).
> But, 128k write performance with VPP is 30 percent lower than the one with kTCP.
> The configuration when VPP starts is as below.
> Is there anyone why 128k write performance with VPP is not good as kernel ?
>
> I increase num-rx-desc and there is improvment, but it still lower than kernel.
> Thanks in advance.
>
> unix {
>         nodaemon
>         cli-listen /run/vpp/cli.sock
>
>         # cli-listen localhost:5002
>
>           full-coredump
>
>
> }
> session {
>         evt_qs_memfd_seg
> }
> dpdk {
>         dev 0000:03:00.0
>         {
>         }
>
> }
> socksvr {
> #       socket-name /tmp/vpp-api.sock
>         socket-name /run/vpp-api.sock
> }
> plugins {
>         plugin default { disable }
>         plugin dpdk_plugin.so { enable }
> }
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org