Re: functional IO in a SHA1 implementation

> Date: Wed, 6 Feb 2019 00:39:09 +0000
> From: <[email protected]>
> To: <[email protected]>
> Subject: [MLton-user] functional IO in a SHA1 implementation
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset="UTF-8"
> 
> There may be better implementations out there, but looking for a SHA1 in SML, I found
> 
>  [email protected]:srdqty/sml-sha1.git?
> 
> It compiles and runs and gives correct answers on at least two inputs (
> It also has a nice generic interface that allows for reading chunks of files at a time, etc. 
> 
> However, the interface is 
> 
>    type 'a byte_reader = 'a * int -> Word8Vector.vector * 'a
>    val sha1 : 'a byte_reader -> 'a -> Word8Vector.vector
> 
> and the realisation of this byte_reader in the supplied sha1-file example is BinIO.StreamIO.inputN. 
> 
> It doesn't look as if the sources are holding onto old states, but, unfortunately, when I run sha1-file on a 1.2GB file, the running process takes much longer than the built-in shasum and it eventually comes to consume about 2.5GB of memory (as eyeballed in OSX's process monitor app). 
> 
> Is this inevitable?  Should I rewrite to make the interface implicitly stateful by just having the byte_reader type be 
> 
>   int -> Word8Vector.vector
> 
> ?
> 
> Michael 

The imperative I/O (i.e., TextIO & BinIO) operations are just layered on top of the
functional stream operations, so the functional layer should be slightly faster.
More significant is the granularity of I/O operations.  The best I/O performance
will come from doing I/O in chunks (e.g., using the input/inputN/inputAll functions).
If the byte_reader type is doing input a byte at a time (using input1), there will
be a lot more overhead.

	- John