Re: [SPDK] Can't create large Malloc bdev
"Harris, James R" <[email protected]> Mon, 10 Apr 2023 23:08:55 +0000
| Newsgroups | dev.linux.lists.spdk |
|---|---|
| Message-ID | <BYAPR11MB33037305B20ABE561275251BDE959@BYAPR11MB3303.namprd11.prod.outlook.com> |
--_000_BYAPR11MB33037305B20ABE561275251BDE959BYAPR11MB3303namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Luis, DPDK has a default limit of 8192 memsegs per list (you probably see some ki= nd of error about this on the target console). A couple of ways I=92m aware of that should work: 1. Patch dpdk/config/rte_config.h locally, increasing RTE_MAX_MEMSEG_PER= _LIST to something larger. The current 8192 default should enable malloc b= devs slightly smaller than 16GiB. 2. Use 1GB hugepages. -Jim From: SPDK <[email protected]> on behalf of Luis Navar= ro <[email protected]> Date: Monday, April 10, 2023 at 3:39 PM To: [email protected] <[email protected]> Subject: [SPDK] Can't create large Malloc bdev Hi all, I'm new to SPDK and testing out the SPDK NVMe-oF target with the latest ESX= i 7.x initiator. I've got an Ubuntu 22.04 LTS server with 512 GB RAM and seem to be unable t= o create a Malloc bdev that's larger than 14 GB. Here=92s what I am seeing: $ sudo HUGEMEM=3D102400 scripts/setup.sh $ sudo scripts/rpc.py bdev_malloc_create -b malloc0 16384 512 request: { "num_blocks": 33554432, "block_size": 512, "name": "malloc0", "method": "bdev_malloc_create", "req_id": 1 } Got JSON-RPC error response response: { "code": -12, "message": "Cannot allocate memory" } What am I doing wrong? Thanks! Luis --_000_BYAPR11MB33037305B20ABE561275251BDE959BYAPR11MB3303namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; font-size:10.0pt; font-family:"Calibri",sans-serif;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; font-size:10.0pt; font-family:"Calibri",sans-serif;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:1561482290; mso-list-type:hybrid; mso-list-template-ids:1098307762 67698705 67698713 67698715 67698703 67698= 713 67698715 67698703 67698713 67698715;} @list l0:level1 {mso-level-text:"%1\)"; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l0:level2 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l0:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l0:level4 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l0:level5 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l0:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} @list l0:level7 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l0:level8 {mso-level-number-format:alpha-lower; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} @list l0:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; text-indent:-9.0pt;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --></style> </head> <body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:= break-word"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">Hi Luis,<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">DPDK has a default = limit of 8192 memsegs per list (you probably see some kind of error about t= his on the target console).<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">A couple of ways I= =92m aware of that should work:<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></= span></p> <ol style=3D"margin-top:0in" start=3D"1" type=3D"1"> <li class=3D"MsoListParagraph" style=3D"margin-left:0in;mso-list:l0 level1 = lfo1"><span style=3D"font-size:11.0pt">Patch dpdk/config/rte_config.h local= ly, increasing RTE_MAX_MEMSEG_PER_LIST to something larger. The curre= nt 8192 default should enable malloc bdevs slightly smaller than 16GiB.<o:p></o:p></span></li><li class=3D"MsoListPar= agraph" style=3D"margin-left:0in;mso-list:l0 level1 lfo1"><span style=3D"fo= nt-size:11.0pt">Use 1GB hugepages.<o:p></o:p></span></li></ol> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">-Jim<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></= span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></= span></p> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"mso-margin-top-alt:0in;margin-right:0in;mar= gin-bottom:12.0pt;margin-left:.5in"> <b><span style=3D"font-size:12.0pt;color:black">From: </span></b><span styl= e=3D"font-size:12.0pt;color:black">SPDK <[email protected]= ion.org> on behalf of Luis Navarro <[email protected]><br> <b>Date: </b>Monday, April 10, 2023 at 3:39 PM<br> <b>To: </b>[email protected] <[email protected]= g><br> <b>Subject: </b>[SPDK] Can't create large Malloc bdev<o:p></o:p></span></p> </div> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt">Hi all,<o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt"> <o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt">I'm new to SPDK and testing out the SPDK NVMe-oF target with the la= test ESXi 7.x initiator.<o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt"> <o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt">I've got an Ubuntu 22.04 LTS server with 512 GB RAM and seem to be = unable to create a Malloc bdev that's larger than 14 GB.<o:p></o:p></span><= /p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt"> <o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt">Here=92s what I am seeing:<o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt"> <o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">$ sudo HUGEMEM=3D102400 scripts/setup.sh</span><= span style=3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">$ sudo scripts/rpc.py bdev_malloc_create -b mall= oc0 16384 512</span><span style=3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">request:</span><span style=3D"font-size:11.0pt">= <o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">{</span><span style=3D"font-size:11.0pt"><o:p></= o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New""> "num_blocks": 33554432,</span><= span style=3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New""> "block_size": 512,</span><span = style=3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New""> "name": "malloc0",</s= pan><span style=3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New""> "method": "bdev_malloc_cre= ate",</span><span style=3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New""> "req_id": 1</span><span style= =3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">}</span><span style=3D"font-size:11.0pt"><o:p></= o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">Got JSON-RPC error response</span><span style=3D= "font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">response:</span><span style=3D"font-size:11.0pt"= ><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">{</span><span style=3D"font-size:11.0pt"><o:p></= o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New""> "code": -12,</span><span style= =3D"font-size:11.0pt"><o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New""> "message": "Cannot allocat= e memory"</span><span style=3D"font-size:11.0pt"><o:p></o:p></span></p= > <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-famil= y:"Courier New"">}</span><span style=3D"font-size:11.0pt"><o:p></= o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt"> <o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt">What am I doing wrong?<o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt"> <o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt">Thanks!<o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt">Luis<o:p></o:p></span></p> <p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:= 11.0pt"> <o:p></o:p></span></p> </div> </body> </html> --_000_BYAPR11MB33037305B20ABE561275251BDE959BYAPR11MB3303namp_--