[metapost] reading a large array

[Hans Hagen]

On 3/10/2017 1:56 AM, Qiong Cai wrote:
> Hi,
>
> Here's the code to read an array of data from a file.
>
> vardefreadFromFile(suffixa)(textfilename) =
>     numericc;
>     c := 0;
>     strings;
>     forever:
>         s := readfrom filename;
>         exitifs = EOF;
>         a[c] := s;
>         c := c+1;
>     endfor
>     c
>
> enddef;
>
>
> The performance of this function on my iMac (3.xGHz, Intel SKL) is shown
> below
>
> 10 numbers: 0.096 seconds
>
> 100: 0.096
>
> 1K: 0.10
>
> 10K: 0.463
>
> 100K: 1m23seconds
>
> 1M: after 15 minutes, still not finishing
>
> It seems that from 100K to 1M, it's getting slower every 10K or 1K. I
> guess the memory allocator in metapost pre-allocates certain memory and
> copies the whole memory when the array increases.  Is that the case?  If
> so, could we improve the memory allocation in metapost?

\starttext

\startluacode
     local t = { }
     for i=1,10000 do
         t[i] =
             "here we have number " ..
             tostring(i) ..
             " out of the 10000 numbers that we will test"
     end
     t = table.concat(t,"\n")
     print(#t)
     io.savedata("foo1.tmp",t)
     io.savedata("foo2.tmp",t)
\stopluacode

\startMPcode
     string f ; f := "foo1.tmp" ;
     string s[] ;
     numeric n ; n := 0 ;
     for i=1 upto 10000 :
         s[i] := readfrom f ;
         exitif s[i] = EOF ;
         n := n + 1 ;
     endfor ;
     message("read from foo1: " & decimal n);
\stopMPcode

\startMPcode
     string f ; f := "foo2.tmp" ;
     string s[] ;
     string ss ;
     numeric n ; n := 0 ;
     for i=1 upto 10000 :
         ss := readfrom f ;
         exitif ss = EOF ;
         s[i] := ss ;
         n := n + 1 ;
     endfor ;
     message("read from foo2: " & decimal n);
\stopMPcode

\stoptext

filesize: 678.893

first test  : 0.79 sec
second test : 0.41 sec

so, given all the string allocation and tokenization that mp has to do 
probably ok

for ten times as long strings the timings are 1.15 and  0.69 seconds and 
for 100 times longer strings (filesize 66.899.399 bytes) 3.1 and 2.5 
seconds so maybe something messy with your system

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------