Re: reading a large array

On 3/10/2017 1:56 AM, Qiong Cai wrote:
> Hi,
>
> Here's the code to read an array of data from a file.
>
> vardefreadFromFile(suffixa)(textfilename) =
>     numericc;
>     c := 0;
>     strings;
>     forever:
>         s := readfrom filename;
>         exitifs = EOF;
>         a[c] := s;
>         c := c+1;
>     endfor
>     c
>
> enddef;
>
>
> The performance of this function on my iMac (3.xGHz, Intel SKL) is shown
> below
>
> 10 numbers: 0.096 seconds
>
> 100: 0.096
>
> 1K: 0.10
>
> 10K: 0.463
>
> 100K: 1m23seconds
>
> 1M: after 15 minutes, still not finishing
>
> It seems that from 100K to 1M, it's getting slower every 10K or 1K. I
> guess the memory allocator in metapost pre-allocates certain memory and
> copies the whole memory when the array increases.  Is that the case?  If
> so, could we improve the memory allocation in metapost?

in context one can do this (normally done cleaner but this shows the 
principle):

\startMPcode{doublefun}
     lua("GlobalData = string.splitlines(io.loaddata('foo3.tmp'))") ;
     numeric l ;
     for i=1 step 1 until lua("mp.print(\#GlobalData)") :
         l := length(lua("mp.quoted(\#GlobalData[" & decimal i & "])")) ;
       %  message(decimal i & ":" & decimal l);
     endfor ;
\stopMPcode

or somewhat nicer:

\startluacode
     local MyData = { }
     function mp.LoadMyData(filename)
         MyData = string.splitlines(io.loaddata(filename))
     end
     function mp.MyDataSize()
         mp.print(#MyData)
     end
     function mp.MyDataString(i)
         mp.quoted(MyData[i] or "")
     end
\stopluacode

\startMPcode{doublefun}
     lua.mp.LoadMyData("foo3.tmp") ;
     numeric l ;
     for i=1 step 1 until lua.mp.MyDataSize() :
         l := length(lua.mp.MyDataString(i)) ;
       % message(decimal i & ":" & decimal l);
     endfor ;
\stopMPcode




this runs in 3 sec on my machine which is way faster than storing at the 
mp end where at some point you hit limitations in constructing names for 
variables (keep in mind that a[i] is not an array but a hashed variable 
name constructed from a and i)

Hans
-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
--
http://tug.org/metapost/