[SPDK] Long IO latency with NVMe/TCP target due to large low watermark socket setting

Hi,

We opened an issue https://github.com/spdk/spdk/issues/1377 but did not see response, so I thought I should bring it here.

In SPDK NVMe/TCP target, when initializing the socket, the low watermark is set to sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 24 bytes. In our testing, some times there might be very small data packet (as small as 16 bytes) be sent to wire. After this, if there is no more data sent to the same socket, this small data packet won’t be received by NVMe/TCP controller qpair thread because the size hasn’t reached the low watermark. Because of this, the qpair thread is waiting for more data come in and the initiator is waiting for the IO request to be completed. Hence the delay happens.

As the minimum data that allows target to determine the PDU type is sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 8 bytes, we changed low watermark setting as below. With the change, the problem was gone immediately.


*** tcp.c.new   2020-04-30 05:31:37.196499792 +0000

--- tcp.c       2020-04-29 19:48:25.857651523 +0000

*************** spdk_nvmf_tcp_qpair_sock_init(struct spd

*** 911,917 ****

        int rc;



        /* set low water mark */

!       rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_common_pdu_hdr));

        if (rc != 0) {

                SPDK_ERRLOG("spdk_sock_set_recvlowat() failed\n");

                return rc;

--- 911,917 ----

        int rc;



        /* set low water mark */

!       rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_c2h_data_hdr));

        if (rc != 0) {

                SPDK_ERRLOG("spdk_sock_set_recvlowat() failed\n");

                return rc;

I would suggest SPDK have this change included in future release.

Thanks,
-Wenhua Liu