[SPDK] Re: Performance impact with "enable_zerocopy_send"

Charlie,

Could you clarify where did you enable ZCOPY - on target side or on initiator side?
If you enable it on target side then it is expected that Read (Controller to Host) has higher IOPS since in Read target writes data to a socket and ZCOPY takes effect on write operation. In Write (Host to Controller) target reads data from a socket and only writes responses back to the initiator.

Best regards,
Alexey Marchuk

-----Original Message-----
From: Charlie Li <charlie.li(a)gmail.com> 
Sent: Wednesday, March 3, 2021 7:28 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: Performance impact with "enable_zerocopy_send"

Hi Alexey,

Looks like you are right that ZEROCOPY is not effective for smaller buffer sizes like 4KB.

Here are the performance changes I got after enabling ZEROCOPY

Random Read in 4KB:                      Dropped  from 5042 KIOPS to 3080
KIOPS; -38.91%
Random Write in 4KB:                      Dropped  from 6301 KIOPS to 5899
KIOPS; -6.38%

Sequential Read in 128KB:               Increased  from 158.3 KIOPS to 262
KIOPS; +65.51%
Sequential Write in 128KB:               Increased  from 161.6 KIOPS to
167.5 KIOPS; +3.65%

Why does XEROCOPY have a much bigger impact on Read than on Write?

Thanks,
Charlie

On Mon, Feb 8, 2021 at 11:37 AM Alexey Marchuk <alexeymar(a)nvidia.com> wrote:

> Hello Charlie,
>
> Could you add more details about the performance drop? Are you using 
> SPDK target and/or initiator and what is payload size?
> I would expect some degradation with small payloads, as written it 
> this doc 
> https://www.kernel.org/doc/html/v4.15/networking/msg_zerocopy.html#opp
> ortunity-and-caveats
> :
> "MSG_ZEROCOPY is generally only effective at writes over around 10 KB. "
>
> Best regards,
> Alexey Marchuk
>
> -----Original Message-----
> From: Charlie Li <charlie.li(a)gmail.com>
> Sent: Wednesday, January 20, 2021 12:41 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Performance impact with "enable_zerocopy_send"
>
> Hello,
>
> In SPDK 20.10, "enable_zerocopy_send" is set to "false" by default.
>
> I tried to set  "enable_zerocopy_send" to true, but noticed the 
> performance dropped.
>
> Is this as expected? My understanding is that setting 
> "enable_zerocopy_send" to true should improve performance.
>
> Thanks,
> Charlie
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org
>
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org