Max array size
Nicholas Jankowski <[email protected]> Sat, 2 Nov 2024 11:39:49 -0400
| Newsgroups | gmane.comp.gnu.octave.general |
|---|---|
| Message-ID | <CALW4+bJqOYwmWeQU2h+kmWDV0Z_yb6es=_+u8RkU8DW2mzZO7A@mail.gmail.com> |
--===============2632056220288226704== Content-Type: multipart/alternative; boundary="000000000000b060840625efdc3e" --000000000000b060840625efdc3e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable As the note says at the bottom of these messages, the help email list is not used much anymore. Active support has moved to the discussion forum at octave.discourse.group , you will likely receive more thorough assistance there. That said, there are maximum array dimension limits in Octave (2^32 =3D 4.2= e9 elements or 2^64 =3D 1.8e19 elements, depending on which windows installer you chose). The 1e12 vector would hit that limit triggering the error message you showed unless you used the w64-64 installer. Despite this, however, a 1e9 vector of 8 byte doubles will consume ~8GB of memory. A 1e12 vector will try to consume ~8TB of memory, exceeding what is available. If you require the ability to work with such large data sets, you will probably need to specially handle such large arrays to work within available memory, swap, and disc storage to do so. I do not know of any builtin functions or external packages in octave made for this, but I=E2=80= =99ll again refer you to the general octave forum at octave.discourse.group where experts may have better advice/suggestions On Fri, Nov 1, 2024, 00:10 DuyKy Nguyen <[email protected]> wrote: > Hello > > My info is on my personal page > > http://unitedthc.com/ > > > > I'm evaluating my new AI structure using OCtave 9.1 on my w7-64b with 32 = G > RAM > > it's happy with rand(1m 1e9) > > but with rand(1, 1e12) > > is say > > error: out of memory or dimension too large for Octave's index type > > so is it possible to huge raaray to evaluate huge neuron number > > Thnks a lot > > DuyKy Nguyen PhD EE > > retired Ex R&D SymmetricomEng > > > > I can use 4 million neuron but > > ---------- > We are transitioning to a web based forum > for community help discussions at > https://octave.discourse.group/c/help > --000000000000b060840625efdc3e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">As the note says at the bottom of these messages, the help e= mail list is not used much anymore.=C2=A0 Active support has moved to the d= iscussion forum at octave.discourse.group , you will likely receive more th= orough assistance there.</p> <p dir=3D"ltr">That said, there are maximum array dimension limits in Octav= e (2^32 =3D 4.2e9 elements or 2^64 =3D =C2=A01.8e19 elements, depending on = which windows installer you chose). The 1e12 vector would hit that limit tr= iggering the error message you showed unless you used the w64-64 installer.= =C2=A0 Despite this, however, a 1e9 vector of 8 byte doubles will consume ~= 8GB of memory. A 1e12 vector will try to consume ~8TB of memory, exceeding = what is available.=C2=A0 If you require the ability to work with such large= data sets, you will probably need to specially handle such large arrays to= work within available memory, swap, and disc storage to do so.=C2=A0 I do = not know of any builtin functions or external packages in octave made for t= his, but I=E2=80=99ll again refer you to the general octave forum at octave= .discourse.group where experts may have better advice/suggestions<br> </p> <br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Fri= , Nov 1, 2024, 00:10 DuyKy Nguyen <<a href=3D"mailto:[email protected]" = target=3D"_blank">[email protected]</a>> wrote:<br></div><blockquote cla= ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa= dding-left:1ex"> <div lang=3D"EN-US" link=3D"blue" vlink=3D"purple"> <div> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">Hello<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">My info is on my personal page<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif""><a href=3D"http://unitedthc.com/" rel=3D"noreferrer"= target=3D"_blank">http://unitedthc.com/</a><u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif""><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">I'm evaluating my new AI structure using OCtave 9.1 on my w7-64b with 32 G RAM<= u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">it's happy with rand(1m 1e9)<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">but with rand(1, 1e12)<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">is say <u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">error: out of memory or dimension too large for Octave's index type<u></u><u><= /u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">so is it possible to huge raaray to evaluate huge neuron number <u></u><u></u></s= pan></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">Thnks a lot<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">DuyKy Nguyen PhD EE<u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">retired Ex R&D SymmetricomEng <u></u><u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif""><u></u>=C2=A0<u></u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif""><img width=3D"450" height=3D"210"></span><span style= =3D"font-family:"Arial Black","sans-serif""><u></u><u><= /u></span></p> <p class=3D"MsoNormal"><span style=3D"font-family:"Arial Black",&= quot;sans-serif"">I can use 4 million neuron but <u></u><u></u></span></p> </div> </div> <br> ----------<br> We are transitioning to a web based forum<br> for community help discussions at<br> <a href=3D"https://octave.discourse.group/c/help" rel=3D"noreferrer norefer= rer" target=3D"_blank">https://octave.discourse.<wbr>group/c/help</a><br> </blockquote></div> --000000000000b060840625efdc3e-- --===============2632056220288226704== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline Ci0tLS0tLS0tLS0KV2UgYXJlIHRyYW5zaXRpb25pbmcgdG8gYSB3ZWIgYmFzZWQgZm9ydW0KZm9y IGNvbW11bml0eSBoZWxwIGRpc2N1c3Npb25zIGF0Cmh0dHBzOi8vb2N0YXZlLmRpc2NvdXJzZS5n cm91cC9jL2hlbHAK --===============2632056220288226704==--