[SPDK] Re: spdk_nvme_ns_cmd_read returning -ENOMEM

Hi Trevor,

Most SSDs don't support anything close to an 8GB read.  MDTS (Max Data Transfer Size) is more typically 128KB-1MB.  The SPDK nvme driver will split requests that exceed this limit, but only to a point.  If you're using an Intel NVMe SSD with a 128KB MDTS, that 8GB read would need to be split into 64K separate commands to the SSD, each of which is represented by an nvme_request structure returned from the nvme_allocate_request() function that you mention.  If the payload size exceeds this limit, you'll get -ENOMEM returned.

You may be able to get it to work if you override the default io_queue_requests when allocating your IO qpair.  You could also split the 8GB read into smaller chunks.

Regards,

-Jim


On 4/24/20, 1:13 PM, "trevordkramer(a)gmail.com" <trevordkramer(a)gmail.com> wrote:

    Hello,

    I have a system with 8 1GB hugepages configured at boot time split between 2 sockets. I reserve all 8 for spdk with 

            struct spdk_env_opts opts{};
            spdk_env_opts_init(&opts);
            opts.mem_size = 1024 * 8;

    When I call spdk_nvme_ns_cmd_read I get -ENOMEM if my buffer (which I allocated with spdk_zmalloc) is too large. The call to spdk_zmalloc succeeds so the buffer is created. I don't understand what the buffer size would have to do with spdk_nvme_ns_cmd_read failing. Can anyone shed any light? I've looked though the code a bit and it looks like nvme_allocate_request must be failing but I don't understand why.

    Thanks,

    Trevor
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org