Re: File I/O Metrics

--000000000000574baa05eb91fb22
Content-Type: text/plain; charset="UTF-8"

I tried changing (make-array buffer-size :element-type 'character)
to
(make-array buffer-size :element-type 'byte)
and I got additional warnings and it took 70 seconds instead of 20.

Thanks,
Garrett.

On Fri, Oct 21, 2022 at 1:47 PM Robert Goldman <[email protected]> wrote:

> I don't know what data you are reading but is there any chance that the
> difference is that when you read text in lisp as ISO-8859-1 lisp is
> actually processing the text as unicode, but when you are reading it in
> Java you are just slamming raw bytes into memory?
>
> Maybe this is relevant?
> https://stackoverflow.com/questions/979932/read-unicode-text-files-with-java
>
> I don't use Java myself, so I can't say, and I don't have access to your
> data, but it does seem like the Java code is doing something simpler than
> the Lisp code.
>
> What happens if you change your Lisp code to read-sequence of type byte
> instead of character?
>
> On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:
>
> I don't want to cause a firestore here but I was doing some simple
> benchmarks on file i/o between Java, ABCL, and SBCL and I'm a bit shocked,
> honestly.
>
> Reading a 2.5M file in 16M chunks in (using iso-8859-1):
> - abcl takes a tad over 1 second
> - sbcl takes 0.04 seconds
>
> Reading a 5.8G file in 16M chunks in (using iso-8859-1 for Lisp, for Java
> it's just bytes):
> - abcl takes...too long, I gave up
> - sbcl takes between 20 and 21 seconds
> - Java takes 1.5 seconds
>
> These are all run on the same computer using the same files, etc.
>
> What's up with this? Thoughts? I'd heard that SBCL should be as fast as C
> under at least some circumstances. I'd wager that C is at least as fast as
> Java (probably faster).
>
> Thanks,
> Garrett Dangerfield. (he/him/his)
>
> P.S. Don't get me wrong, I *LOVE* Lisp, I'm trying to get away from Java
> as
> fast as I can (the syntax is killing me slowly). I've used ABCL in
> projects before (it was wonderful, Java doesn't handle XML well).
>
> Lisp code:
> (with-open-file (stream "/media/danger/OS/temp/jars.txt" :external-format
> :iso-8859-1) ; great_expectations.iso
> (let ((size (file-length stream))
> (buffer-size (* 16 1024 1024)) ; 16M
> )
> (time
> (loop with buffer = (make-array buffer-size :element-type 'character)
> for n-characters = (read-sequence buffer stream)
> while (< 0 n-characters)))
> )))
>
> Java code:
> private static final int BUFFER_SIZE = 16 * 1024 * 1024;
> try (InputStream in = new
> FileInputStream("/media/danger/OS/temp/great_expectations.iso"); ) {
> byte[] buff = new byte[BUFFER_SIZE];
> int chunkLen = -1;
> long start = System.currentTimeMillis();
> while ((chunkLen = in.read(buff)) != -1) {
> System.out.println("chunkLen = " + chunkLen);
> }
> double duration = System.currentTimeMillis() - start;
> duration /= 1000;
> System.out.println(String.format("it took %,2f secs", duration));
> } catch (Exception e) {
> e.printStackTrace(System.out);
> } finally {
> System.out.println("Done.");
> }
>
> Robert P. Goldman
> Research Fellow
> Smart Information Flow Technologies (d/b/a SIFT, LLC)
>
> 319 N. First Ave., Suite 400
> Minneapolis, MN 55401
>
> Voice: (612) 326-3934
> Email: [email protected]
>

--000000000000574baa05eb91fb22
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>I tried changing (make-array buffer-size :element-typ=
e &#39;character)</div><div>to</div><div>(make-array buffer-size :element-t=
ype &#39;byte)</div><div>and I got additional warnings and it took 70 secon=
ds instead of 20.</div><div><br></div><div>Thanks,</div><div>Garrett.<br></=
div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_at=
tr">On Fri, Oct 21, 2022 at 1:47 PM Robert Goldman &lt;<a href=3D"mailto:rp=
[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid=
 rgb(204,204,204);padding-left:1ex"><u></u>




<div><div style=3D"font-family:sans-serif"><div style=3D"white-space:normal=
">
<p dir=3D"auto">I don&#39;t know what data you are reading but is there any=
 chance that the difference is that when you read text in lisp as ISO-8859-=
1 lisp is actually processing the text as unicode, but when you are reading=
 it in Java you are just slamming raw bytes into memory?</p>
<p dir=3D"auto">Maybe this is relevant? <a href=3D"https://stackoverflow.co=
m/questions/979932/read-unicode-text-files-with-java" style=3D"color:rgb(57=
,131,196)" target=3D"_blank">https://stackoverflow.com/questions/979932/rea=
d-unicode-text-files-with-java</a></p>
<p dir=3D"auto">I don&#39;t use Java myself, so I can&#39;t say, and I don&=
#39;t have access to your data, but it does seem like the Java code is doin=
g something simpler than the Lisp code.</p>
<p dir=3D"auto">What happens if you change your Lisp code to <code style=3D=
"margin:0px;padding:0px 0.25em;border-radius:3px;background-color:rgb(247,2=
47,247)">read-sequence</code> of type <code style=3D"margin:0px;padding:0px=
 0.25em;border-radius:3px;background-color:rgb(247,247,247)">byte</code> in=
stead of <code style=3D"margin:0px;padding:0px 0.25em;border-radius:3px;bac=
kground-color:rgb(247,247,247)">character</code>?</p>
<p dir=3D"auto">On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:</p>
</div><div style=3D"white-space:normal"><blockquote style=3D"margin:0px 0px=
 5px;padding-left:5px;border-left:2px solid rgb(119,119,119);color:rgb(119,=
119,119)"><p dir=3D"auto">I don&#39;t want to cause a firestore here but I =
was doing some simple
<br>
benchmarks on file i/o between Java, ABCL, and SBCL and I&#39;m a bit shock=
ed,
<br>
honestly.</p>
<p dir=3D"auto">Reading a 2.5M file in 16M chunks in (using iso-8859-1):
<br>
- abcl takes a tad over 1 second
<br>
- sbcl takes 0.04 seconds</p>
<p dir=3D"auto">Reading a 5.8G file in 16M chunks in (using iso-8859-1 for =
Lisp, for Java
<br>
it&#39;s just bytes):
<br>
- abcl takes...too long, I gave up
<br>
- sbcl takes between 20 and 21 seconds
<br>
- Java takes 1.5 seconds</p>
<p dir=3D"auto">These are all run on the same computer using the same files=
, etc.</p>
<p dir=3D"auto">What&#39;s up with this?  Thoughts?  I&#39;d heard that SBC=
L should be as fast as C
<br>
under at least some circumstances.  I&#39;d wager that C is at least as fas=
t as
<br>
Java (probably faster).</p>
<p dir=3D"auto">Thanks,
<br>
Garrett Dangerfield. (he/him/his)</p>
<p dir=3D"auto">P.S. Don&#39;t get me wrong, I *LOVE* Lisp, I&#39;m trying =
to get away from Java as
<br>
fast as I can (the syntax is killing me slowly).  I&#39;ve used ABCL in
<br>
projects before (it was wonderful, Java doesn&#39;t handle XML well).</p>
<p dir=3D"auto">Lisp code:
<br>
  (with-open-file (stream &quot;/media/danger/OS/temp/jars.txt&quot; :exter=
nal-format
<br>
:iso-8859-1) ; great_expectations.iso
<br>
 (let ((size (file-length stream))
<br>
(buffer-size (* 16 1024 1024)) ; 16M
<br>
)
<br>
   (time
<br>
    (loop with buffer =3D (make-array buffer-size :element-type &#39;charac=
ter)
<br>
  for n-characters =3D (read-sequence buffer stream)
<br>
  while (&lt; 0 n-characters)))
<br>
   )))</p>
<p dir=3D"auto">Java code:
<br>
private static final int BUFFER_SIZE =3D 16 * 1024 * 1024;
<br>
try (InputStream in =3D new
<br>
FileInputStream(&quot;/media/danger/OS/temp/great_expectations.iso&quot;); =
) {
<br>
byte[] buff =3D new byte[BUFFER_SIZE];
<br>
int chunkLen =3D -1;
<br>
long start =3D System.currentTimeMillis();
<br>
while ((chunkLen =3D in.read(buff)) !=3D -1) {
<br>
System.out.println(&quot;chunkLen =3D &quot; + chunkLen);
<br>
}
<br>
double duration =3D System.currentTimeMillis() - start;
<br>
duration /=3D 1000;
<br>
System.out.println(String.format(&quot;it took %,2f secs&quot;, duration));
<br>
} catch (Exception e) {
<br>
e.printStackTrace(System.out);
<br>
} finally {
<br>
System.out.println(&quot;Done.&quot;);
<br>
}</p>
</blockquote></div>
<div style=3D"white-space:normal">
<p dir=3D"auto">Robert P. Goldman<br>
Research Fellow<br>
Smart Information Flow Technologies (d/b/a SIFT, LLC)</p>
<p dir=3D"auto">319 N. First Ave., Suite 400<br>
Minneapolis, MN 55401</p>
<p dir=3D"auto">Voice:	(612) 326-3934<br>
Email:    <a href=3D"mailto:[email protected]" style=3D"color:rgb(57,131,1=
96)" target=3D"_blank">[email protected]</a></p>

</div>
</div>
</div>


</blockquote></div>

--000000000000574baa05eb91fb22--