[SPDK] Re: Long IO latency with NVMe/TCP target due to large low watermark socket setting

Hi  Wenhua,

You are right. If we set this value very large, it will cause latency if there is small packet comming.  I checked the code and spec,  the initiator will send 4 kind of PDU,  icreq, h2cdata, Capsucmd and h2cterm.  So after checking the spec, the smallest pdu size is 16. And it should be set to command pdu header (which is 8 bytes) or just set to 16 bytes (it is also OK.). If possible, could you directly submit a patch to spdk using the development guides in spdk.io?

Thanks.


Best Regards
Ziye Yang 

-----Original Message-----
From: Wenhua Liu <liuw(a)vmware.com> 
Sent: Thursday, April 30, 2020 2:51 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Long IO latency with NVMe/TCP target due to large low watermark socket setting

Hi,

We opened an issue https://github.com/spdk/spdk/issues/1377 but did not see response, so I thought I should bring it here.

In SPDK NVMe/TCP target, when initializing the socket, the low watermark is set to sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 24 bytes. In our testing, some times there might be very small data packet (as small as 16 bytes) be sent to wire. After this, if there is no more data sent to the same socket, this small data packet won’t be received by NVMe/TCP controller qpair thread because the size hasn’t reached the low watermark. Because of this, the qpair thread is waiting for more data come in and the initiator is waiting for the IO request to be completed. Hence the delay happens.

As the minimum data that allows target to determine the PDU type is sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 8 bytes, we changed low watermark setting as below. With the change, the problem was gone immediately.


*** tcp.c.new   2020-04-30 05:31:37.196499792 +0000

--- tcp.c       2020-04-29 19:48:25.857651523 +0000

*************** spdk_nvmf_tcp_qpair_sock_init(struct spd

*** 911,917 ****

        int rc;



        /* set low water mark */

!       rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_common_pdu_hdr));

        if (rc != 0) {

                SPDK_ERRLOG("spdk_sock_set_recvlowat() failed\n");

                return rc;

--- 911,917 ----

        int rc;



        /* set low water mark */

!       rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_c2h_data_hdr));

        if (rc != 0) {

                SPDK_ERRLOG("spdk_sock_set_recvlowat() failed\n");

                return rc;

I would suggest SPDK have this change included in future release.

Thanks,
-Wenhua Liu
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org