Re: functional IO in a SHA1 implementation
Matthew Fluet <[email protected]> Wed, 6 Feb 2019 10:59:06 -0500
| Newsgroups | gmane.comp.lang.ml.mlton.user |
|---|---|
| Message-ID | <CAMrhFL5z20QGPq=g=g77YPywVAdpi_q0WLoVOycU3hbnAbdruw@mail.gmail.com> |
--===============0256519592400693917== Content-Type: multipart/alternative; boundary="000000000000e16fdd05813bce9e" --000000000000e16fdd05813bce9e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Feb 5, 2019 at 8:09 PM <[email protected]> wrote: > There may be better implementations out there, but looking for a SHA1 in > SML, I found > > [email protected]:srdqty/sml-sha1.git=EF=BB=BF > > It compiles and runs and gives correct answers on at least two inputs ( > It also has a nice generic interface that allows for reading chunks of > files at a time, etc. > > However, the interface is > > type 'a byte_reader =3D 'a * int -> Word8Vector.vector * 'a > val sha1 : 'a byte_reader -> 'a -> Word8Vector.vector > > and the realisation of this byte_reader in the supplied sha1-file example > is BinIO.StreamIO.inputN. > > It doesn't look as if the sources are holding onto old states, but, > unfortunately, when I run sha1-file on a 1.2GB file, the running process > takes much longer than the built-in shasum and it eventually comes to > consume about 2.5GB of memory (as eyeballed in OSX's process monitor app)= . > > Is this inevitable? Should I rewrite to make the interface implicitly > stateful by just having the byte_reader type be > > int -> Word8Vector.vector > > ? > Although the SHA1 computation doesn't hold onto old states, the `sha1-file.sml` driver does: val fstream =3D BinIO.openIn filename val fstream' =3D BinIO.getInstream fstream val hash =3D SHA1.sha1String BinIO.StreamIO.inputN fstream' val _ =3D print (hash ^ " " ^ filename ^ "\n") val _ =3D BinIO.closeIn fstream The `fstream` retained to close the file is a reference to the beginning of the file, so that is keeping the whole stream alive through the SHA1 computation. For later comparison, here is the behavior of the original program (on a 1.8G file): mtf@oxygen examples]$ ./sha1-file @MLton gc-summary -- Fedora-Workstation-Live-x86_64-29-1.2.iso 34b8316e4bae6620edbb97b45aacf63ceb5d9294 Fedora-Workstation-Live-x86_64-29-1.2.iso GC type time ms number bytes bytes/sec ------------- ------- ------- --------------- --------------- copying 656 9 962,790,424 1,467,668,292 mark-compact 0 0 0 - minor 0 0 0 - total time: 22,872 ms total GC time: 740 ms (3.2%) max pause time: 520 ms total bytes allocated: 6,394,284,784 bytes max bytes live: 656,971,664 bytes max heap size: 5,255,847,936 bytes max stack size: 912 bytes num cards marked: 0 bytes scanned: 0 bytes bytes hash consed: 0 bytes Note the low number of GCs and the high max heap size. max bytes live is not quite 1.8G, because it is the maximum live observed at a GC (not the maximum live at any point in the program); MLton resizes the heap at each GC to be (approximately) 8x live data. So, the program kept filling the heap and GCing until about half the file was loaded; that corresponds to the 656K max bytes live, at which point the heap was resized to 8 * 656K = =3D 5.2G (the max heap size), which sufficed for loading the rest of the file and finishing without another GC (and max bytes live sample). One simple solution is to not explicitly close the stream; MLton simply allows the file to close when the process exits. Commenting out the `val _ =3D BinIO.closeIn fstream` gives the following behavior: [mtf@oxygen examples]$ ./sha1-file @MLton gc-summary -- Fedora-Workstation-Live-x86_64-29-1.2.iso 34b8316e4bae6620edbb97b45aacf63ceb5d9294 Fedora-Workstation-Live-x86_64-29-1.2.iso GC type time ms number bytes bytes/sec ------------- ------- ------- --------------- --------------- copying 144 21,764 528,256,200 3,668,445,777 mark-compact 0 0 0 - minor 0 0 0 - total time: 20,084 ms total GC time: 180 ms (0.9%) max pause time: 4 ms total bytes allocated: 6,412,906,064 bytes max bytes live: 24,664 bytes max heap size: 319,488 bytes max stack size: 816 bytes num cards marked: 0 bytes scanned: 0 bytes bytes hash consed: 0 bytes Now the program runs in constant heap space. However, the total time isn't significantly faster (20s vs 22s; and much higher than the 3.7s that sha1sum takes); we do a lot of GCs (because there is little live data and the heap stays small, but rapidly fills), but that doesn't seem to be the dominant cost. Moreover, presumably you are looking to incorporate this into a larger SML program and probably don't want to leave open file handles. With MLton (more below), you can replace `val _ =3D BinIO.closeIn fstream` with `val _= =3D BinIO.StreamIO.closeIn fstream'` to get the following behavior: [mtf@oxygen examples]$ ./sha1-file @MLton gc-summary -- Fedora-Workstation-Live-x86_64-29-1.2.iso 34b8316e4bae6620edbb97b45aacf63ceb5d9294 Fedora-Workstation-Live-x86_64-29-1.2.iso GC type time ms number bytes bytes/sec ------------- ------- ------- --------------- --------------- copying 160 21,795 538,295,824 3,364,348,800 mark-compact 0 0 0 - minor 0 0 0 - total time: 20,096 ms total GC time: 196 ms (1.0%) max pause time: 4 ms total bytes allocated: 6,414,327,600 bytes max bytes live: 28,464 bytes max heap size: 319,488 bytes max stack size: 880 bytes num cards marked: 0 bytes scanned: 0 bytes bytes hash consed: 0 bytes That gives the same constant heap space. One might be confused why keeping `fstream'` (the "obvious" reference to the beginning of the stream) live retains less data than keeping `fstream` (the non-obvious reference to the beginning of the stream) live. This is due to MLton optimizations and a MLton's implementation of `BinIO.instream`. Basically, MLton is able to optimize `BinIO.StreamIO.closeIn fstream'` to only keep the primitive reader component of the stream alive; it doesn't need the chain of buffers. However, in general, one might not be able to rely on that kind of optimization. I don't know if other SML systems would do so. A somewhat portable, but not necessarily robust, solution would be to extract the close function from the primitive reader from the `BinIO.Stream.instream` at the time that `fstream'` is created and then call that function after the SHA1 computation. That should avoid keeping the beginning of the stream alive through the SHA1 comptuation. However, that might violate some invariants of the Basis Library implementation; for example, the `BinIO.Stream.instream` implementation might have a flag to know that it's been closed, which won't be properly set by calling the primitive reader's close function. The proper solution would be for the SHA1 computation to return the final state of the stream. Then one would write: val fstream =3D BinIO.openIn filename val fstream' =3D BinIO.getInstream fstream val (hash, fstream'') =3D SHA1.sha1String BinIO.StreamIO.inputN fstream= ' val _ =3D print (hash ^ " " ^ filename ^ "\n") val _ =3D BinIO.StreamIO.closeIn fstream'' But, in any case, it doesn't look like the memory usage and GC is the dominant cost of the computation. It might actually be faster to shell out to `sha1sum`, possibly using pipes if the data to be hashed is in memory, rather than in files. -Matthew --=20 You received this message because you are subscribed to the Google Groups "= MLton-user" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to [email protected]. --000000000000e16fdd05813bce9e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div di= r=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" st= yle=3D"font-family:"courier new",monospace;font-size:large"><span= style=3D"font-family:Arial,Helvetica,sans-serif;font-size:small">On Tue, F= eb 5, 2019 at 8:09 PM <<a href=3D"mailto:[email protected]= ">[email protected]</a>> wrote:</span><br></div></div><div= class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0p= x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Th= ere may be better implementations out there, but looking for a SHA1 in SML,= I found<br> <br> =C2=A0 [email protected]:srdqty/sml-sha1.git=EF=BB=BF<br> <br> It compiles and runs and gives correct answers on at least two inputs (<br> It also has a nice generic interface that allows for reading chunks of file= s at a time, etc. <br> <br> However, the interface is <br> <br> =C2=A0 =C2=A0 type 'a byte_reader =3D 'a * int -> Word8Vector.ve= ctor * 'a<br> =C2=A0 =C2=A0 val sha1 : 'a byte_reader -> 'a -> Word8Vector.= vector<br> <br> and the realisation of this byte_reader in the supplied sha1-file example i= s BinIO.StreamIO.inputN. <br> <br> It doesn't look as if the sources are holding onto old states, but, unf= ortunately, when I run sha1-file on a 1.2GB file, the running process takes= much longer than the built-in shasum and it eventually comes to consume ab= out 2.5GB of memory (as eyeballed in OSX's process monitor app). <br> <br> Is this inevitable?=C2=A0 Should I rewrite to make the interface implicitly= stateful by just having the byte_reader type be <br> <br> =C2=A0 =C2=A0int -> Word8Vector.vector<br> <br> ?<br></blockquote><div><br></div><div class=3D"gmail_default" style=3D"font= -family:"courier new",monospace;font-size:large">Although the SHA= 1 computation doesn't hold onto old states, the `sha1-file.sml` driver = does:</div><div class=3D"gmail_default" style=3D"font-family:"courier = new",monospace;font-size:large"><br></div><div class=3D"gmail_default"= ><span style=3D"font-family:"courier new",monospace;font-size:lar= ge">=C2=A0 =C2=A0 </span><font face=3D"courier new, monospace" size=3D"4">v= al fstream =3D BinIO.openIn filename</font></div><div class=3D"gmail_defaul= t"><font face=3D"courier new, monospace" size=3D"4">=C2=A0 =C2=A0 val fstre= am' =3D BinIO.getInstream fstream</font></div><div class=3D"gmail_defau= lt"><span style=3D"font-family:"courier new",monospace;font-size:= large">=C2=A0 =C2=A0 val hash =3D SHA1.sha1String BinIO.StreamIO.inputN fst= ream'</span><br></div><div class=3D"gmail_default"><span style=3D"font-= family:"courier new",monospace;font-size:large">=C2=A0 =C2=A0 val= _ =3D print (hash ^ "=C2=A0 " ^ filename ^ "\n")</span= ><br></div><div class=3D"gmail_default"><span style=3D"font-family:"co= urier new",monospace;font-size:large">=C2=A0 =C2=A0 val _ =3D BinIO.cl= oseIn fstream</span><br></div><div class=3D"gmail_default"><span style=3D"f= ont-family:"courier new",monospace;font-size:large"><br></span></= div><div class=3D"gmail_default"><span style=3D"font-family:"courier n= ew",monospace;font-size:large">The `fstream` retained to close the fil= e is a reference to the beginning of the file, so that is keeping the whole= stream alive through the SHA1 computation.=C2=A0 For later comparison, her= e is the behavior of the original program (on a 1.8G file):</span></div><di= v class=3D"gmail_default"><span style=3D"font-family:"courier new"= ;,monospace;font-size:large"><br></span></div><div class=3D"gmail_default">= <div class=3D"gmail_default" style=3D"font-family:"courier new",m= onospace;font-size:large">mtf@oxygen examples]$ ./sha1-file @MLton gc-summa= ry -- Fedora-Workstation-Live-x86_64-29-1.2.iso<br></div><div class=3D"gmai= l_default" style=3D"font-family:"courier new",monospace;font-size= :large">34b8316e4bae6620edbb97b45aacf63ceb5d9294=C2=A0 Fedora-Workstation-L= ive-x86_64-29-1.2.iso</div><div class=3D"gmail_default" style=3D"font-famil= y:"courier new",monospace;font-size:large">GC type=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0time ms=C2=A0 number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0bytes=C2=A0 =C2=A0 =C2=A0 =C2=A0bytes/sec</div><div class=3D"gmail_de= fault" style=3D"font-family:"courier new",monospace;font-size:lar= ge">-------------=C2=A0 =C2=A0------- ------- --------------- -------------= --</div><div class=3D"gmail_default" style=3D"font-family:"courier new= ",monospace;font-size:large">copying=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0656=C2=A0 =C2=A0 =C2=A0 =C2=A09=C2=A0 =C2=A0 =C2=A0962,790,42= 4=C2=A0 =C2=A01,467,668,292</div><div class=3D"gmail_default" style=3D"font= -family:"courier new",monospace;font-size:large">mark-compact=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0-</div><div class=3D"gmail_default" style=3D"font-f= amily:"courier new",monospace;font-size:large">minor=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2= =A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-</div><div class=3D"gmail_default= " style=3D"font-family:"courier new",monospace;font-size:large">t= otal time: 22,872 ms</div><div class=3D"gmail_default" style=3D"font-family= :"courier new",monospace;font-size:large">total GC time: 740 ms (= 3.2%)</div><div class=3D"gmail_default" style=3D"font-family:"courier = new",monospace;font-size:large">max pause time: 520 ms</div><div class= =3D"gmail_default" style=3D"font-family:"courier new",monospace;f= ont-size:large">total bytes allocated: 6,394,284,784 bytes</div><div class= =3D"gmail_default" style=3D"font-family:"courier new",monospace;f= ont-size:large">max bytes live: 656,971,664 bytes</div><div class=3D"gmail_= default" style=3D"font-family:"courier new",monospace;font-size:l= arge">max heap size: 5,255,847,936 bytes</div><div class=3D"gmail_default" = style=3D"font-family:"courier new",monospace;font-size:large">max= stack size: 912 bytes</div><div class=3D"gmail_default" style=3D"font-fami= ly:"courier new",monospace;font-size:large">num cards marked: 0</= div><div class=3D"gmail_default" style=3D"font-family:"courier new&quo= t;,monospace;font-size:large">bytes scanned: 0 bytes</div><div class=3D"gma= il_default" style=3D"font-family:"courier new",monospace;font-siz= e:large">bytes hash consed: 0 bytes</div><div class=3D"gmail_default" style= =3D"font-family:"courier new",monospace;font-size:large"><br></di= v><div class=3D"gmail_default" style=3D"font-family:"courier new"= ,monospace;font-size:large">Note the low number of GCs and the high max hea= p size.=C2=A0 max bytes live is not quite 1.8G, because it is the maximum l= ive observed at a GC (not the maximum live at any point in the program); ML= ton resizes the heap at each GC to be (approximately) 8x live data.=C2=A0 S= o, the program kept filling the heap and GCing until about half the file wa= s loaded; that corresponds to the 656K max bytes live, at which point the h= eap was resized to 8 * 656K =3D 5.2G (the max heap size), which sufficed fo= r loading the rest of the file and finishing without another GC (and max by= tes live sample).</div><div class=3D"gmail_default" style=3D"font-family:&q= uot;courier new",monospace;font-size:large"><br></div><div class=3D"gm= ail_default" style=3D"font-family:"courier new",monospace;font-si= ze:large">One simple solution is to not explicitly close the stream; MLton = simply allows the file to close when the process exits.=C2=A0 Commenting ou= t the `val _ =3D BinIO.closeIn fstream` gives the following behavior:</div>= <div class=3D"gmail_default" style=3D"font-family:"courier new",m= onospace;font-size:large"><br></div><div class=3D"gmail_default" style=3D""= ><div class=3D"gmail_default" style=3D"font-family:"courier new",= monospace;font-size:large">[mtf@oxygen examples]$ ./sha1-file @MLton gc-sum= mary -- Fedora-Workstation-Live-x86_64-29-1.2.iso</div><div class=3D"gmail_= default" style=3D"font-family:"courier new",monospace;font-size:l= arge">34b8316e4bae6620edbb97b45aacf63ceb5d9294=C2=A0 Fedora-Workstation-Liv= e-x86_64-29-1.2.iso</div><div class=3D"gmail_default" style=3D"font-family:= "courier new",monospace;font-size:large">GC type=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0time ms=C2=A0 number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0bytes=C2=A0 =C2=A0 =C2=A0 =C2=A0bytes/sec</div><div class=3D"gmail_defau= lt" style=3D"font-family:"courier new",monospace;font-size:large"= >-------------=C2=A0 =C2=A0------- ------- --------------- ---------------<= /div><div class=3D"gmail_default" style=3D"font-family:"courier new&qu= ot;,monospace;font-size:large">copying=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0144=C2=A0 21,764=C2=A0 =C2=A0 =C2=A0528,256,200=C2=A0 =C2=A03,= 668,445,777</div><div class=3D"gmail_default" style=3D"font-family:"co= urier new",monospace;font-size:large">mark-compact=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0-</div><div class=3D"gmail_default" style=3D"font-family:"courie= r new",monospace;font-size:large">minor=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0-</div><div class=3D"gmail_default" style=3D"font-f= amily:"courier new",monospace;font-size:large">total time: 20,084= ms</div><div class=3D"gmail_default" style=3D"font-family:"courier ne= w",monospace;font-size:large">total GC time: 180 ms (0.9%)</div><div c= lass=3D"gmail_default" style=3D"font-family:"courier new",monospa= ce;font-size:large">max pause time: 4 ms</div><div class=3D"gmail_default" = style=3D"font-family:"courier new",monospace;font-size:large">tot= al bytes allocated: 6,412,906,064 bytes</div><div class=3D"gmail_default" s= tyle=3D"font-family:"courier new",monospace;font-size:large">max = bytes live: 24,664 bytes</div><div class=3D"gmail_default" style=3D"font-fa= mily:"courier new",monospace;font-size:large">max heap size: 319,= 488 bytes</div><div class=3D"gmail_default" style=3D"font-family:"cour= ier new",monospace;font-size:large">max stack size: 816 bytes</div><di= v class=3D"gmail_default" style=3D"font-family:"courier new",mono= space;font-size:large">num cards marked: 0</div><div class=3D"gmail_default= " style=3D"font-family:"courier new",monospace;font-size:large">b= ytes scanned: 0 bytes</div><div class=3D"gmail_default" style=3D"font-famil= y:"courier new",monospace;font-size:large">bytes hash consed: 0 b= ytes</div><div class=3D"gmail_default" style=3D"font-family:"courier n= ew",monospace;font-size:large"><br></div><div class=3D"gmail_default" = style=3D"font-family:"courier new",monospace;font-size:large">Now= the program runs in constant heap space.=C2=A0 However, the total time isn= 't significantly faster (20s vs 22s; and much higher than the 3.7s that= sha1sum takes); we do a lot of GCs (because there is little live data and = the heap stays small, but rapidly fills), but that doesn't seem to be t= he dominant cost.</div><div class=3D"gmail_default" style=3D"font-family:&q= uot;courier new",monospace;font-size:large"><br></div><div class=3D"gm= ail_default" style=3D"font-family:"courier new",monospace;font-si= ze:large">Moreover, presumably you are looking to incorporate this into a l= arger SML program and probably don't want to leave open file handles.= =C2=A0 With MLton (more below), you can replace `val _ =3D BinIO.closeIn fs= tream` with `val _ =3D BinIO.StreamIO.closeIn fstream'` to get the foll= owing behavior:</div><div class=3D"gmail_default" style=3D"font-family:&quo= t;courier new",monospace;font-size:large"><br></div><div class=3D"gmai= l_default" style=3D""><div class=3D"gmail_default" style=3D"font-family:&qu= ot;courier new",monospace;font-size:large">[mtf@oxygen examples]$ ./sh= a1-file @MLton gc-summary -- Fedora-Workstation-Live-x86_64-29-1.2.iso</div= ><div class=3D"gmail_default" style=3D"font-family:"courier new",= monospace;font-size:large">34b8316e4bae6620edbb97b45aacf63ceb5d9294=C2=A0 F= edora-Workstation-Live-x86_64-29-1.2.iso</div><div class=3D"gmail_default" = style=3D"font-family:"courier new",monospace;font-size:large">GC = type=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0time ms=C2=A0 number=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0bytes=C2=A0 =C2=A0 =C2=A0 =C2=A0bytes/sec</div><div= class=3D"gmail_default" style=3D"font-family:"courier new",monos= pace;font-size:large">-------------=C2=A0 =C2=A0------- ------- -----------= ---- ---------------</div><div class=3D"gmail_default" style=3D"font-family= :"courier new",monospace;font-size:large">copying=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0160=C2=A0 21,795=C2=A0 =C2=A0 =C2=A0538,2= 95,824=C2=A0 =C2=A03,364,348,800</div><div class=3D"gmail_default" style=3D= "font-family:"courier new",monospace;font-size:large">mark-compac= t=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0-</div><div class=3D"gmail_default" style=3D"fon= t-family:"courier new",monospace;font-size:large">minor=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 = =C2=A00=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-</div><div class=3D"gmail_def= ault" style=3D"font-family:"courier new",monospace;font-size:larg= e">total time: 20,096 ms</div><div class=3D"gmail_default" style=3D"font-fa= mily:"courier new",monospace;font-size:large">total GC time: 196 = ms (1.0%)</div><div class=3D"gmail_default" style=3D"font-family:"cour= ier new",monospace;font-size:large">max pause time: 4 ms</div><div cla= ss=3D"gmail_default" style=3D"font-family:"courier new",monospace= ;font-size:large">total bytes allocated: 6,414,327,600 bytes</div><div clas= s=3D"gmail_default" style=3D"font-family:"courier new",monospace;= font-size:large">max bytes live: 28,464 bytes</div><div class=3D"gmail_defa= ult" style=3D"font-family:"courier new",monospace;font-size:large= ">max heap size: 319,488 bytes</div><div class=3D"gmail_default" style=3D"f= ont-family:"courier new",monospace;font-size:large">max stack siz= e: 880 bytes</div><div class=3D"gmail_default" style=3D"font-family:"c= ourier new",monospace;font-size:large">num cards marked: 0</div><div c= lass=3D"gmail_default" style=3D"font-family:"courier new",monospa= ce;font-size:large">bytes scanned: 0 bytes</div><div class=3D"gmail_default= " style=3D"font-family:"courier new",monospace;font-size:large">b= ytes hash consed: 0 bytes</div><div style=3D"font-family:"courier new&= quot;,monospace;font-size:large"><br></div><div style=3D"font-family:"= courier new",monospace;font-size:large">That gives the same constant h= eap space.=C2=A0 One might be confused why keeping `fstream'` (the &quo= t;obvious" reference to the beginning of the stream) live retains less= data than keeping `fstream` (the non-obvious reference to the beginning of= the stream) live.=C2=A0 This is due to MLton optimizations and a MLton'= ;s implementation of `BinIO.instream`.=C2=A0 Basically, MLton is able to op= timize `BinIO.StreamIO.closeIn fstream'` to only keep the primitive rea= der component of the stream alive; it doesn't need the chain of buffers= .=C2=A0 However, in general, one might not be able to rely on that kind of = optimization.=C2=A0 I don't know if other SML systems would do so.</div= ><div style=3D"font-family:"courier new",monospace;font-size:larg= e"><br></div><div style=3D"font-family:"courier new",monospace;fo= nt-size:large">A somewhat portable, but not necessarily robust, solution wo= uld be to extract the close function from the primitive reader from the `Bi= nIO.Stream.instream`=C2=A0at the time that `fstream'` is created and th= en call that function after the SHA1 computation.=C2=A0 That should avoid k= eeping the beginning of the stream alive through the SHA1 comptuation.=C2= =A0 However, that might violate some invariants of the Basis Library implem= entation; for example, the `BinIO.Stream.instream` implementation might hav= e a flag to know that it's been closed, which won't be properly set= by calling the primitive reader's close function.</div><div style=3D"f= ont-family:"courier new",monospace;font-size:large"><br></div><di= v style=3D"font-family:"courier new",monospace;font-size:large">T= he proper solution would be for the SHA1 computation to return the final st= ate of the stream.=C2=A0 Then one would write:</div><div style=3D"font-fami= ly:"courier new",monospace;font-size:large"><br></div><div style= =3D""><div class=3D"gmail_default" style=3D"font-family:Arial,Helvetica,san= s-serif;font-size:small"><span style=3D"font-family:"courier new"= ,monospace;font-size:large">=C2=A0 =C2=A0 </span><font face=3D"courier new,= monospace" size=3D"4">val fstream =3D BinIO.openIn filename</font></div><d= iv class=3D"gmail_default" style=3D"font-family:Arial,Helvetica,sans-serif;= font-size:small"><font face=3D"courier new, monospace" size=3D"4">=C2=A0 = =C2=A0 val fstream' =3D BinIO.getInstream fstream</font></div><div clas= s=3D"gmail_default" style=3D"font-family:Arial,Helvetica,sans-serif;font-si= ze:small"><span style=3D"font-family:"courier new",monospace;font= -size:large">=C2=A0 =C2=A0 val (hash, fstream'') =3D SHA1.sha1Strin= g BinIO.StreamIO.inputN fstream'</span><br></div><div class=3D"gmail_de= fault" style=3D"font-family:Arial,Helvetica,sans-serif;font-size:small"><sp= an style=3D"font-family:"courier new",monospace;font-size:large">= =C2=A0 =C2=A0 val _ =3D print (hash ^ "=C2=A0 " ^ filename ^ &quo= t;\n")</span><br></div><div class=3D"gmail_default" style=3D"font-fami= ly:Arial,Helvetica,sans-serif;font-size:small"><span style=3D"font-family:&= quot;courier new",monospace;font-size:large">=C2=A0 =C2=A0 val _ =3D B= inIO.StreamIO.closeIn fstream''</span><br></div><div class=3D"gmail= _default" style=3D"font-family:Arial,Helvetica,sans-serif;font-size:small">= <span style=3D"font-family:"courier new",monospace;font-size:larg= e"><br></span></div><div class=3D"gmail_default" style=3D""><font face=3D"c= ourier new, monospace" size=3D"4">But, in any case, it doesn't look lik= e the memory usage and GC is the dominant cost of the computation.=C2=A0 It= might actually be faster to shell out to `sha1sum`, possibly using pipes i= f the data to be hashed is in memory, rather than in files.</font></div><di= v class=3D"gmail_default" style=3D""><font face=3D"courier new, monospace" = size=3D"4"><br></font></div><div class=3D"gmail_default" style=3D""><font f= ace=3D"courier new, monospace" size=3D"4">-Matthew</font></div><div class= =3D"gmail_default" style=3D""><font face=3D"courier new, monospace" size=3D= "4"><br></font></div><div class=3D"gmail_default" style=3D"font-family:Aria= l,Helvetica,sans-serif;font-size:small"><br></div></div><div class=3D"gmail= _default" style=3D"font-family:Arial,Helvetica,sans-serif;font-size:small">= <br></div></div></div></div></div></div></div></div></div></div></div> <p></p> -- <br /> You received this message because you are subscribed to the Google Groups &= quot;MLton-user" group.<br /> To unsubscribe from this group and stop receiving emails from it, send an e= mail to <a href=3D"mailto:[email protected]">mlton-user+unsu= [email protected]</a>.<br /> --000000000000e16fdd05813bce9e-- --===============0256519592400693917== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============0256519592400693917== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ MLton-user mailing list [email protected]; [email protected] https://lists.sourceforge.net/lists/listinfo/mlton-user --===============0256519592400693917==--