[SPDK] Re: spdk_nvme_ns_cmd_read returning -ENOMEM

On Sat, Apr 25, 2020, 00:18 Trevor Kramer <trevordkramer(a)gmail.com> wrote:

> Thanks Jim. I should have been more specific. The 8GB is just the hugepages
> allocated. The read buffer size is 25MB - 50MB. A 25MB read works and
> anything larger fails but I assume your point is still valid.
>

I guess the culprit here is that ENOMEM return is kind of misleading. In
SPDK, it usually means that the downstream driver is out of some
memory-backed resource, not the actual RAM which may be a plenty, as in
your case.

The resource in question here is, most likely, nvme_requests which are
statically preallocated based on the io_queue_requests as Jim has pointed
out. The latter seems to be 256 in your setup, which suffices to break 25MB
into 128K requests and fails for 50MB. Increasing queue depth should
resolve the issue.

Regards,
Andrey

>
> Thanks,
>
> Trevor
>
>
> On Fri, Apr 24, 2020 at 3:22 PM Harris, James R <james.r.harris(a)intel.com>
> wrote:
>
> > Hi Trevor,
> >
> > Most SSDs don't support anything close to an 8GB read.  MDTS (Max Data
> > Transfer Size) is more typically 128KB-1MB.  The SPDK nvme driver will
> > split requests that exceed this limit, but only to a point.  If you're
> > using an Intel NVMe SSD with a 128KB MDTS, that 8GB read would need to be
> > split into 64K separate commands to the SSD, each of which is represented
> > by an nvme_request structure returned from the nvme_allocate_request()
> > function that you mention.  If the payload size exceeds this limit,
> you'll
> > get -ENOMEM returned.
> >
> > You may be able to get it to work if you override the default
> > io_queue_requests when allocating your IO qpair.  You could also split
> the
> > 8GB read into smaller chunks.
> >
> > Regards,
> >
> > -Jim
> >
> >
> > On 4/24/20, 1:13 PM, "trevordkramer(a)gmail.com" <trevordkramer(a)gmail.com
> >
> > wrote:
> >
> >     Hello,
> >
> >     I have a system with 8 1GB hugepages configured at boot time split
> > between 2 sockets. I reserve all 8 for spdk with
> >
> >             struct spdk_env_opts opts{};
> >             spdk_env_opts_init(&opts);
> >             opts.mem_size = 1024 * 8;
> >
> >     When I call spdk_nvme_ns_cmd_read I get -ENOMEM if my buffer (which I
> > allocated with spdk_zmalloc) is too large. The call to spdk_zmalloc
> > succeeds so the buffer is created. I don't understand what the buffer
> size
> > would have to do with spdk_nvme_ns_cmd_read failing. Can anyone shed any
> > light? I've looked though the code a bit and it looks like
> > nvme_allocate_request must be failing but I don't understand why.
> >
> >     Thanks,
> >
> >     Trevor
> >     _______________________________________________
> >     SPDK mailing list -- spdk(a)lists.01.org
> >     To unsubscribe send an email to spdk-leave(a)lists.01.org
> >
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> >
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>