Re: File I/O Metrics
Robert Goldman <[email protected]> Fri, 21 Oct 2022 15:47:04 -0500
| Newsgroups | gmane.editors.j.devel |
|---|---|
| Message-ID | <[email protected]> |
--=_MailMate_3825CF73-E4CD-4BC4-8B5A-58B2D8B25C37_=
Content-Type: text/plain; format=flowed
I don't know what data you are reading but is there any chance that the
difference is that when you read text in lisp as ISO-8859-1 lisp is
actually processing the text as unicode, but when you are reading it in
Java you are just slamming raw bytes into memory?
Maybe this is relevant?
https://stackoverflow.com/questions/979932/read-unicode-text-files-with-java
I don't use Java myself, so I can't say, and I don't have access to your
data, but it does seem like the Java code is doing something simpler
than the Lisp code.
What happens if you change your Lisp code to `read-sequence` of type
`byte` instead of `character`?
On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:
> I don't want to cause a firestore here but I was doing some simple
> benchmarks on file i/o between Java, ABCL, and SBCL and I'm a bit
> shocked,
> honestly.
>
> Reading a 2.5M file in 16M chunks in (using iso-8859-1):
> - abcl takes a tad over 1 second
> - sbcl takes 0.04 seconds
>
> Reading a 5.8G file in 16M chunks in (using iso-8859-1 for Lisp, for
> Java
> it's just bytes):
> - abcl takes...too long, I gave up
> - sbcl takes between 20 and 21 seconds
> - Java takes 1.5 seconds
>
> These are all run on the same computer using the same files, etc.
>
> What's up with this? Thoughts? I'd heard that SBCL should be as fast
> as C
> under at least some circumstances. I'd wager that C is at least as
> fast as
> Java (probably faster).
>
> Thanks,
> Garrett Dangerfield. (he/him/his)
>
> P.S. Don't get me wrong, I *LOVE* Lisp, I'm trying to get away from
> Java as
> fast as I can (the syntax is killing me slowly). I've used ABCL in
> projects before (it was wonderful, Java doesn't handle XML well).
>
> Lisp code:
> (with-open-file (stream "/media/danger/OS/temp/jars.txt"
> :external-format
> :iso-8859-1) ; great_expectations.iso
> (let ((size (file-length stream))
> (buffer-size (* 16 1024 1024)) ; 16M
> )
> (time
> (loop with buffer = (make-array buffer-size :element-type
> 'character)
> for n-characters = (read-sequence buffer stream)
> while (< 0 n-characters)))
> )))
>
> Java code:
> private static final int BUFFER_SIZE = 16 * 1024 * 1024;
> try (InputStream in = new
> FileInputStream("/media/danger/OS/temp/great_expectations.iso"); ) {
> byte[] buff = new byte[BUFFER_SIZE];
> int chunkLen = -1;
> long start = System.currentTimeMillis();
> while ((chunkLen = in.read(buff)) != -1) {
> System.out.println("chunkLen = " + chunkLen);
> }
> double duration = System.currentTimeMillis() - start;
> duration /= 1000;
> System.out.println(String.format("it took %,2f secs", duration));
> } catch (Exception e) {
> e.printStackTrace(System.out);
> } finally {
> System.out.println("Done.");
> }
Robert P. Goldman
Research Fellow
Smart Information Flow Technologies (d/b/a SIFT, LLC)
319 N. First Ave., Suite 400
Minneapolis, MN 55401
Voice: (612) 326-3934
Email: [email protected]
--=_MailMate_3825CF73-E4CD-4BC4-8B5A-58B2D8B25C37_=
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html>
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"=
>
</head>
<body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty=
le=3D"white-space: normal;">
<p dir=3D"auto">I don't know what data you are reading but is there any c=
hance that the difference is that when you read text in lisp as ISO-8859-=
1 lisp is actually processing the text as unicode, but when you are readi=
ng it in Java you are just slamming raw bytes into memory?</p>
<p dir=3D"auto">Maybe this is relevant? <a href=3D"https://stackoverflow.=
com/questions/979932/read-unicode-text-files-with-java" style=3D"color: #=
3983C4;">https://stackoverflow.com/questions/979932/read-unicode-text-fil=
es-with-java</a></p>
<p dir=3D"auto">I don't use Java myself, so I can't say, and I don't have=
access to your data, but it does seem like the Java code is doing someth=
ing simpler than the Lisp code.</p>
<p dir=3D"auto">What happens if you change your Lisp code to <code style=3D=
"margin: 0 0; padding: 0 0.25em; border-radius: 3px; background-color: #F=
7F7F7;">read-sequence</code> of type <code style=3D"margin: 0 0; padding:=
0 0.25em; border-radius: 3px; background-color: #F7F7F7;">byte</code> in=
stead of <code style=3D"margin: 0 0; padding: 0 0.25em; border-radius: 3p=
x; background-color: #F7F7F7;">character</code>?</p>
<p dir=3D"auto">On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:</p>
</div><div class=3D"plaintext" style=3D"white-space: normal;"><blockquote=
style=3D"margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #777=
777; color: #777777;"><p dir=3D"auto">I don't want to cause a firestore h=
ere but I was doing some simple
<br>
benchmarks on file i/o between Java, ABCL, and SBCL and I'm a bit shocked=
,
<br>
honestly.</p>
<p dir=3D"auto">Reading a 2.5M file in 16M chunks in (using iso-8859-1):
<br>
- abcl takes a tad over 1 second
<br>
- sbcl takes 0.04 seconds</p>
<p dir=3D"auto">Reading a 5.8G file in 16M chunks in (using iso-8859-1 fo=
r Lisp, for Java
<br>
it's just bytes):
<br>
- abcl takes...too long, I gave up
<br>
- sbcl takes between 20 and 21 seconds
<br>
- Java takes 1.5 seconds</p>
<p dir=3D"auto">These are all run on the same computer using the same fil=
es, etc.</p>
<p dir=3D"auto">What's up with this? Thoughts? I'd heard that SBCL shou=
ld be as fast as C
<br>
under at least some circumstances. I'd wager that C is at least as fast =
as
<br>
Java (probably faster).</p>
<p dir=3D"auto">Thanks,
<br>
Garrett Dangerfield. (he/him/his)</p>
<p dir=3D"auto">P.S. Don't get me wrong, I *LOVE* Lisp, I'm trying to get=
away from Java as
<br>
fast as I can (the syntax is killing me slowly). I've used ABCL in
<br>
projects before (it was wonderful, Java doesn't handle XML well).</p>
<p dir=3D"auto">Lisp code:
<br>
(with-open-file (stream "/media/danger/OS/temp/jars.txt" :external-form=
at
<br>
:iso-8859-1) ; great_expectations.iso
<br>
(let ((size (file-length stream))
<br>
(buffer-size (* 16 1024 1024)) ; 16M
<br>
)
<br>
(time
<br>
(loop with buffer =3D (make-array buffer-size :element-type 'characte=
r)
<br>
for n-characters =3D (read-sequence buffer stream)
<br>
while (< 0 n-characters)))
<br>
)))</p>
<p dir=3D"auto">Java code:
<br>
private static final int BUFFER_SIZE =3D 16 * 1024 * 1024;
<br>
try (InputStream in =3D new
<br>
FileInputStream("/media/danger/OS/temp/great_expectations.iso"); ) {
<br>
byte[] buff =3D new byte[BUFFER_SIZE];
<br>
int chunkLen =3D -1;
<br>
long start =3D System.currentTimeMillis();
<br>
while ((chunkLen =3D in.read(buff)) !=3D -1) {
<br>
System.out.println("chunkLen =3D " + chunkLen);
<br>
}
<br>
double duration =3D System.currentTimeMillis() - start;
<br>
duration /=3D 1000;
<br>
System.out.println(String.format("it took %,2f secs", duration));
<br>
} catch (Exception e) {
<br>
e.printStackTrace(System.out);
<br>
} finally {
<br>
System.out.println("Done.");
<br>
}</p>
</blockquote></div>
<div class=3D"markdown" style=3D"white-space: normal;">
<p dir=3D"auto">Robert P. Goldman<br>
Research Fellow<br>
Smart Information Flow Technologies (d/b/a SIFT, LLC)</p>
<p dir=3D"auto">319 N. First Ave., Suite 400<br>
Minneapolis, MN 55401</p>
<p dir=3D"auto">Voice: (612) 326-3934<br>
Email: <a href=3D"mailto:[email protected]" style=3D"color: #3983C4;"=
>[email protected]</a></p>
</div>
</div>
</body>
</html>
--=_MailMate_3825CF73-E4CD-4BC4-8B5A-58B2D8B25C37_=--