[SPDK] Re: Single iSCSI max performance with SPDK NULL Block Device

Hi, Andrey:

Current target server setting about Linux
1. Disable irqbalance, set CPU scaling governor to performance mode
2. TCP configuration
        tcp_configs = [{"file": "/proc/sys/net/ipv4/tcp_timestamps",
"value": "1"},
                       {"file": "/proc/sys/net/ipv4/tcp_sack", "value":
"0"},
                       {"file": "/proc/sys/net/ipv4/tcp_rmem", "value":
"10000000 10000000 10000000"},
                       {"file": "/proc/sys/net/ipv4/tcp_wmem", "value":
"10000000 10000000 10000000"},
                       {"file": "/proc/sys/net/ipv4/tcp_mem", "value":
"10000000 10000000 10000000"},
                       {"file": "/proc/sys/net/core/rmem_default", "value":
"524287"},
                       {"file": "/proc/sys/net/core/wmem_default", "value":
"524287"},
                       {"file": "/proc/sys/net/core/rmem_max", "value":
"524287"},
                       {"file": "/proc/sys/net/core/wmem_max", "value":
"524287"},
                       {"file": "/proc/sys/net/core/optmem_max", "value":
"524287"},
                       {"file": "/proc/sys/net/core/netdev_max_backlog",
"value": "300000"}]
3. NIC Card IRQ affinity : vcore 1, 24, 25
    Storage Target Server: Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz
    Number of CPU : 2 (Total vcores: 48)
4. Enable NIC rps_sock_flow_entries and rps_cpu to 1,24,25 also
5. SPDK iSCSI reactor: QPDefaultCPUMask 0x08007C08007C
6. I can get line rate (1180K, BS=4K, 100%RndW or 100%RndR) if I use >= 8
SPDK NULL Block Devices with >= 8 SPDK iSCSI targets.

On Wed, Dec 9, 2020 at 9:58 PM Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
wrote:

> Ah, you're is ng null bdev, so my reference to the SSD being an issue is
> irrelevant. At 40 Gig, you should be running much faster than  155 KIOPS,
> and the target threading may be an issue. OTOH, single poller with null
> bdev has been shown to reach 10 MIOPS, so I'd also look into your network
> setup.
>
> Regards,
> Andrey
>
> On Wed, Dec 9, 2020, 16:51 Lego Lin <lego.lin(a)gmail.com> wrote:
>
> > Hi,  Andrey
> >
> > Thanks for quick reply. I will try 20.10.
> > Sorry for providing wrong information. Single target with SPDK NULL BDev
> > under 40G nic card, I got
> > 100%RandWrite 4K : 135K
> > 100%RandRead 4K : 155K
> >
> > Although 135K for 1 SSD random write is enough, but not enough for read.
> > And if I compose 4 SSD into 1 raid-0 BDev, 135K is not enough.
> >
> > Anyway, I will try 20.10 first. Thanks again
> >
> > On Wed, Dec 9, 2020 at 8:36 PM Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com>
> > wrote:
> >
> > > On Wed, Dec 9, 2020, 15:14 Lego Lin <lego.lin(a)gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Currently, I am using SPDK 19.04 evaluating SPDK iSCSI Performance.
> As
> > >
> > > I know, current SPDK iSCSI only uses 1 reactor to handle 1 target's
> iSCSI
> > > > IO.
> > > > When I try with SPDK NULL Block Device, the max performance with
> > > > 100%RandWrite / 40G NIC / BS = 4K is about 155K.
> > > >
> > >
> > > At 155K random write IOPS, the bottleneck can easily be in the actual
> SSD
> > > being used. Have you tested w/o SPDK iSCSI target being involved, with
> > the
> > > same SSD in the direct attachment settings?
> > >
> > > Also, 19.04 is 1.5 years old, so you might want to upgrade to 20.10.
> > >
> > > Regards,
> > > Andrey
> > >
> > > I try to finish IO immediately at different layer : SPDK BDev Layer,
> SCSI
> > > > Layer.
> > > > It looks like iSCSI protocol handling become the bottleneck of SPDK
> > NULL
> > > > Block Device.
> > > > Is there any way in SPDK to let multiple reactors handle 1 iSCSI
> target
> > > > IOs?
> > > >
> > > > Current test topology as following:
> > >
> > >
> > > > Client: FIO <--> /dev/sdb (Test 1: with 1 thread + with/without
> blk-mq,
> > > > Test 2: with 4 threads  + with/without blk-mq)
> > > > Target: 1 SPDK iSCSI target <--> 1 LUN <--> SPDK NULL Block Device
> > >
> > >
> > > > Thanks
> > > > _______________________________________________
> > > > SPDK mailing list -- spdk(a)lists.01.org
> > > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> > > >
> > > _______________________________________________
> > > SPDK mailing list -- spdk(a)lists.01.org
> > > To unsubscribe send an email to spdk-leave(a)lists.01.org
> > >
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> >
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>