[SPDK] Re: spdk_nvme_ns_cmd_read returning -ENOMEM

Thanks Jim. I should have been more specific. The 8GB is just the hugepages
allocated. The read buffer size is 25MB - 50MB. A 25MB read works and
anything larger fails but I assume your point is still valid.

Thanks,

Trevor


On Fri, Apr 24, 2020 at 3:22 PM Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Trevor,
>
> Most SSDs don't support anything close to an 8GB read.  MDTS (Max Data
> Transfer Size) is more typically 128KB-1MB.  The SPDK nvme driver will
> split requests that exceed this limit, but only to a point.  If you're
> using an Intel NVMe SSD with a 128KB MDTS, that 8GB read would need to be
> split into 64K separate commands to the SSD, each of which is represented
> by an nvme_request structure returned from the nvme_allocate_request()
> function that you mention.  If the payload size exceeds this limit, you'll
> get -ENOMEM returned.
>
> You may be able to get it to work if you override the default
> io_queue_requests when allocating your IO qpair.  You could also split the
> 8GB read into smaller chunks.
>
> Regards,
>
> -Jim
>
>
> On 4/24/20, 1:13 PM, "trevordkramer(a)gmail.com" <trevordkramer(a)gmail.com>
> wrote:
>
>     Hello,
>
>     I have a system with 8 1GB hugepages configured at boot time split
> between 2 sockets. I reserve all 8 for spdk with
>
>             struct spdk_env_opts opts{};
>             spdk_env_opts_init(&opts);
>             opts.mem_size = 1024 * 8;
>
>     When I call spdk_nvme_ns_cmd_read I get -ENOMEM if my buffer (which I
> allocated with spdk_zmalloc) is too large. The call to spdk_zmalloc
> succeeds so the buffer is created. I don't understand what the buffer size
> would have to do with spdk_nvme_ns_cmd_read failing. Can anyone shed any
> light? I've looked though the code a bit and it looks like
> nvme_allocate_request must be failing but I don't understand why.
>
>     Thanks,
>
>     Trevor
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
>
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>