[SPDK] Re: Print backtrace in SPDK

Hi Wenhua,

You need to compile spdk with --with-uring option.  And you need to 
1 Download the liburing and install it by yourself.
2 Check your kernel version. Uring socket implementation depends on the kernel (> 5.4.3).

What's you kernel version in the VM?

Thanks.




Best Regards
Ziye Yang 

-----Original Message-----
From: Wenhua Liu <liuw(a)vmware.com> 
Sent: Monday, August 24, 2020 3:19 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: Print backtrace in SPDK

Hi Ziye,

I'm using SPDK NVMe-oF target.

I used some other way and figured out the following call path:
posix_sock_group_impl_poll
-> _sock_flush    <------------------ failed
-> spdk_sock_abort_requests
   -> _pdu_write_done
      -> nvmf_tcp_qpair_disconnect
         -> spdk_nvmf_qpair_disconnect
            -> _nvmf_qpair_destroy
               -> spdk_nvmf_poll_group_remove
                  -> nvmf_transport_poll_group_remove
                     -> nvmf_tcp_poll_group_remove
                        -> spdk_sock_group_remove_sock
                           -> posix_sock_group_impl_remove_sock
                              -> spdk_sock_abort_requests
               -> _nvmf_ctrlr_free_from_qpair
                  -> _nvmf_transport_qpair_fini
                     -> nvmf_transport_qpair_fini
                        -> nvmf_tcp_close_qpair
                           -> spdk_sock_close

The _sock_flush calls sendmsg to write the data to the socket. It's sendmsg failing with return value -1. I captured wire data. In Wireshark, I can see the READ command has been received by the target as a TCP packet. As the response to this TCP packet, a TCP packet with FIN flag set is sent to the initiator. The FIN is to close the socket connection.

I'm running SPDK target inside a VM. My NVMe/TCP initiator runs inside another VM. I'm going to try with another SPDK target which runs on a physical machine.

By the way, I noticed there is a uring based sock implementation,  how do I switch to this sock implementation. It seems the default is posix sock implementation.

Thanks,
-Wenhua 

On 8/23/20, 9:55 PM, "Yang, Ziye" <ziye.yang(a)intel.com> wrote:

    Hi Wenhua,

    Which applications are you using from SPDK?  
    1 SPDK NVMe-oF target in target side?
    2  SPDK NVMe perf or others?

    For nvmf_tcp_close_qpair will be called in the following possible cases (not all listed) for TCP transport. But it will be called by spdk_nvmf_qpair_disconnect as the entry.

    1  qpair is not in polling group
    spdk_nvmf_qpair_disconnect
    	nvmf_transport_qpair_fini

    2  spdk_nvmf_qpair_disconnect
    		....
    	_nvmf_qpair_destroy
    		nvmf_transport_qpair_fini
    			..
    			nvmf_tcp_close_qpair


    3  spdk_nvmf_qpair_disconnect
    		....
    	_nvmf_qpair_destroy
    		_nvmf_ctrlr_free_from_qpair	
    			_nvmf_transport_qpair_fini
    				..
    				nvmf_tcp_close_qpair


    spdk_nvmf_qpair_disconnect is called by nvmf_tcp_qpair_disconnect in tcp.c. nvmf_tcp_qpair_disconnect is called in the following cases:

    (1) _pdu_write_done (if there is error for write);
    (2) nvmf_tcp_qpair_handle_timeout.( No response from initiator in 30s if targets sends c2h_term_req)
    (3) nvmf_tcp_capsule_cmd_hdr_handle. (Cannot get tcp req)
    (4) nvmf_tcp_sock_cb.   TCP PDU related handling issue. 


    Also in lib/nvmf/ctrlr.c Target side has a timer poller: nvmf_ctrlr_keep_alive_poll. If there is no keep alive command sent from host, it will call spdk_nvmf_qpair_disconnect in related polling group assoicated with the controller.


    Best Regards
    Ziye Yang 

    -----Original Message-----
    From: Wenhua Liu <liuw(a)vmware.com> 
    Sent: Saturday, August 22, 2020 3:15 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: [SPDK] Print backtrace in SPDK

    Hi,

    Does anyone know if there is a function in SPDK that prints the backtrace?

    I run into a “Connection Reset by Peer” issue on host side when testing NVMe/TCP. I identified it’s because some queue pairs are closed unexpectedly by calling nvmf_tcp_close_qpair, but I could not figure out how/why this function is called. I thought if the backtrace can be printed when calling this function, it might be helpful to me to find the root cause.

    Thanks,
    -Wenhua
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org