Re: [SPDK] Can't create large Malloc bdev
"Luis Navarro" <[email protected]> Thu, 20 Apr 2023 15:51:12 -0700
| Newsgroups | dev.linux.lists.spdk |
|---|---|
| Message-ID | <[email protected]> |
This is a multipart message in MIME format. ------=_NextPart_000_2C3C_01D9739F.EEEFFD80 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi Jim, My apologies for the delay in responding. I tried both approaches and they both worked perfectly! Thanks! Luis From: Harris, James R [mailto:[email protected]] Sent: Monday, April 10, 2023 4:09 PM To: Luis Navarro <[email protected]>; [email protected] Subject: Re: [SPDK] Can't create large Malloc bdev Hi Luis, DPDK has a default limit of 8192 memsegs per list (you probably see some kind of error about this on the target console). A couple of ways I'm aware of that should work: 1) Patch dpdk/config/rte_config.h locally, increasing RTE_MAX_MEMSEG_PER_LIST to something larger. The current 8192 default should enable malloc bdevs slightly smaller than 16GiB. 2) Use 1GB hugepages. -Jim From: SPDK <[email protected]> on behalf of Luis Navarro <[email protected] <mailto:[email protected]> > Date: Monday, April 10, 2023 at 3:39 PM To: [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]> > Subject: [SPDK] Can't create large Malloc bdev Hi all, I'm new to SPDK and testing out the SPDK NVMe-oF target with the latest ESXi 7.x initiator. I've got an Ubuntu 22.04 LTS server with 512 GB RAM and seem to be unable to create a Malloc bdev that's larger than 14 GB. Here's what I am seeing: $ sudo HUGEMEM=102400 scripts/setup.sh $ sudo scripts/rpc.py bdev_malloc_create -b malloc0 16384 512 request: { "num_blocks": 33554432, "block_size": 512, "name": "malloc0", "method": "bdev_malloc_create", "req_id": 1 } Got JSON-RPC error response response: { "code": -12, "message": "Cannot allocate memory" } What am I doing wrong? Thanks! Luis ------=_NextPart_000_2C3C_01D9739F.EEEFFD80 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 = (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:10.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; margin-bottom:.0001pt; font-size:10.0pt; font-family:"Calibri",sans-serif;} span.EmailStyle18 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:windowtext;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:306399739; mso-list-template-ids:-1258121734;} @list l1 {mso-list-id:1561482290; mso-list-type:hybrid; mso-list-template-ids:1098307762 67698705 67698713 67698715 67698703 = 67698713 67698715 67698703 67698713 67698715;} @list l1:level1 {mso-level-text:"%1\)"; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l1:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l1:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l1:level4 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l1:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l1:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l1:level7 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l1:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l1:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-US = link=3D"#0563C1" vlink=3D"#954F72"><div class=3DWordSection1><p = class=3DMsoNormal><span style=3D'font-size:11.0pt;color:#1F497D'>Hi = Jim,<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;color:#1F497D'><o:p> </o:p></span></p><p = class=3DMsoNormal><span style=3D'font-size:11.0pt;color:#1F497D'>My = apologies for the delay in responding. I tried both approaches and = they both worked perfectly!<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;color:#1F497D'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;color:#1F497D'>Thanks!<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;color:#1F497D'>Luis<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;color:#1F497D'><o:p> </o:p></span></p><div= ><div style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt = 0in 0in 0in'><p class=3DMsoNormal><b><span = style=3D'font-size:11.0pt'>From:</span></b><span = style=3D'font-size:11.0pt'> Harris, James R = [mailto:[email protected]] <br><b>Sent:</b> Monday, April 10, = 2023 4:09 PM<br><b>To:</b> Luis Navarro <[email protected]>; = [email protected]<br><b>Subject:</b> Re: [SPDK] Can't = create large Malloc bdev<o:p></o:p></span></p></div></div><p = class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt'>Hi Luis,<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt'><o:p> </o:p></span></p><p = class=3DMsoNormal><span style=3D'font-size:11.0pt'>DPDK has a default = limit of 8192 memsegs per list (you probably see some kind of error = about this on the target console).<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt'><o:p> </o:p></span></p><p = class=3DMsoNormal><span style=3D'font-size:11.0pt'>A couple of ways = I’m aware of that should work:<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt'><o:p> </o:p></span></p><p = class=3DMsoNormal = style=3D'margin-left:.5in;text-indent:-.25in;mso-list:l1 level1 = lfo3'><![if !supportLists]><span style=3D'font-size:11.0pt'><span = style=3D'mso-list:Ignore'>1)<span style=3D'font:7.0pt "Times New = Roman"'> = </span></span></span><![endif]><span style=3D'font-size:11.0pt'>Patch = dpdk/config/rte_config.h locally, increasing RTE_MAX_MEMSEG_PER_LIST to = something larger. The current 8192 default should enable malloc = bdevs slightly smaller than 16GiB.<o:p></o:p></span></p><p = class=3DMsoNormal = style=3D'margin-left:.5in;text-indent:-.25in;mso-list:l1 level1 = lfo3'><![if !supportLists]><span style=3D'font-size:11.0pt'><span = style=3D'mso-list:Ignore'>2)<span style=3D'font:7.0pt "Times New = Roman"'> = </span></span></span><![endif]><span style=3D'font-size:11.0pt'>Use 1GB = hugepages.<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt'>-Jim<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt'><o:p> </o:p></span></p><div = style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in = 0in 0in'><p class=3DMsoNormal = style=3D'mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;mar= gin-left:.5in'><b><span style=3D'font-size:12.0pt;color:black'>From: = </span></b><span style=3D'font-size:12.0pt;color:black'>SPDK <<a = href=3D"mailto:[email protected]">spdk-bounces@lists= .linuxfoundation.org</a>> on behalf of Luis Navarro <<a = href=3D"mailto:[email protected]">[email protected]</a>><br><b>Date: = </b>Monday, April 10, 2023 at 3:39 PM<br><b>To: </b><a = href=3D"mailto:[email protected]">[email protected]= .org</a> <<a = href=3D"mailto:[email protected]">[email protected]= .org</a>><br><b>Subject: </b>[SPDK] Can't create large Malloc = bdev<o:p></o:p></span></p></div><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-size:11.0pt'>Hi = all,<o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'> <o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'>I'm new to SPDK and testing out the SPDK = NVMe-oF target with the latest ESXi 7.x = initiator.<o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'> <o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'>I've got an Ubuntu 22.04 LTS server with 512 = GB RAM and seem to be unable to create a Malloc bdev that's larger than = 14 GB.<o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'> <o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'>Here’s what I am = seeing:<o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'> <o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-family:"Courier New"'>$ sudo HUGEMEM=3D102400 = scripts/setup.sh</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier New"'>$ = sudo scripts/rpc.py bdev_malloc_create -b malloc0 16384 512</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'>request:</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'>{</span><span style=3D'font-size:11.0pt'><o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-family:"Courier New"'> "num_blocks": = 33554432,</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'> "block_size": 512,</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'> "name": "malloc0",</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'> "method": = "bdev_malloc_create",</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'> "req_id": 1</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'>}</span><span style=3D'font-size:11.0pt'><o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-family:"Courier New"'>Got JSON-RPC error = response</span><span style=3D'font-size:11.0pt'><o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-family:"Courier New"'>response:</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'>{</span><span style=3D'font-size:11.0pt'><o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-family:"Courier New"'> "code": = -12,</span><span style=3D'font-size:11.0pt'><o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-family:"Courier New"'> "message": = "Cannot allocate memory"</span><span = style=3D'font-size:11.0pt'><o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span style=3D'font-family:"Courier = New"'>}</span><span style=3D'font-size:11.0pt'><o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'> <o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'>What am I doing = wrong?<o:p></o:p></span></p><p class=3DMsoNormal = style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'> <o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'>Thanks!<o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'>Luis<o:p></o:p></span></p><p = class=3DMsoNormal style=3D'margin-left:.5in'><span = style=3D'font-size:11.0pt'> <o:p></o:p></span></p></div></body></htm= l> ------=_NextPart_000_2C3C_01D9739F.EEEFFD80--