Re: File I/O Metrics
Robert Goldman <[email protected]> Fri, 21 Oct 2022 16:42:35 -0500
| Newsgroups | gmane.editors.j.devel |
|---|---|
| Message-ID | <[email protected]> |
--=_MailMate_D7BF15E5-56CA-4574-9F3C-E92A33073DCE_=
Content-Type: text/plain; format=flowed
I took a file of about 450MB of characters. Using SBCL, when I read it
like this:
```
(defun do-test2 ()
(with-open-file (stream *text-file*)
(let ((buffer-size (* 16 1024 1024)) ; 16M
)
(time
(loop with buffer = (make-array buffer-size
:element-type 'character)
for n-characters = (read-sequence buffer stream)
while (< 0 n-characters))))))
```
It took an average of 1.08125s to read (4 trials).
This procedure:
```
(defun do-test3 ()
(with-open-file (stream *text-file* :element-type
'(unsigned-byte 8))
(let ((buffer-size (* 16 1024 1024)) ; 16M
)
(time
(loop with buffer = (make-array buffer-size
:element-type '(unsigned-byte 8))
for n-characters = (read-sequence buffer stream)
while (< 0 n-characters))))))
```
It took an average of 0.07s
Modifying this to set the `:external-format` to `:iso8859-1` and reading
into an array of `:element-type 'character` it takes an average of
0.8095s
So there seems to be *some* overhead to the unicode handling. Note that
I didn't have a file at hand that actually had ISO8859-1 in it, so I
don't know if that would have complicated matters.
This suggests that just moving around bits without worrying about their
interpretation *may* be faster than treating them as characters. So you
could see if that changes your results at all.
I'm not a real expert in CL file I/O, so it's likely that this could be
done better.
On 21 Oct 2022, at 16:18, Garrett Dangerfield wrote:
> I tried changing (make-array buffer-size :element-type 'character)
> to
> (make-array buffer-size :element-type 'byte)
> and I got additional warnings and it took 70 seconds instead of 20.
>
> Thanks,
> Garrett.
>
> On Fri, Oct 21, 2022 at 1:47 PM Robert Goldman <[email protected]>
> wrote:
>
>> I don't know what data you are reading but is there any chance that
>> the
>> difference is that when you read text in lisp as ISO-8859-1 lisp is
>> actually processing the text as unicode, but when you are reading it
>> in
>> Java you are just slamming raw bytes into memory?
>>
>> Maybe this is relevant?
>> https://stackoverflow.com/questions/979932/read-unicode-text-files-with-java
>>
>> I don't use Java myself, so I can't say, and I don't have access to
>> your
>> data, but it does seem like the Java code is doing something simpler
>> than
>> the Lisp code.
>>
>> What happens if you change your Lisp code to read-sequence of type
>> byte
>> instead of character?
>>
>> On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:
>>
>> I don't want to cause a firestore here but I was doing some simple
>> benchmarks on file i/o between Java, ABCL, and SBCL and I'm a bit
>> shocked,
>> honestly.
>>
>> Reading a 2.5M file in 16M chunks in (using iso-8859-1):
>> - abcl takes a tad over 1 second
>> - sbcl takes 0.04 seconds
>>
>> Reading a 5.8G file in 16M chunks in (using iso-8859-1 for Lisp, for
>> Java
>> it's just bytes):
>> - abcl takes...too long, I gave up
>> - sbcl takes between 20 and 21 seconds
>> - Java takes 1.5 seconds
>>
>> These are all run on the same computer using the same files, etc.
>>
>> What's up with this? Thoughts? I'd heard that SBCL should be as fast
>> as C
>> under at least some circumstances. I'd wager that C is at least as
>> fast as
>> Java (probably faster).
>>
>> Thanks,
>> Garrett Dangerfield. (he/him/his)
>>
>> P.S. Don't get me wrong, I *LOVE* Lisp, I'm trying to get away from
>> Java
>> as
>> fast as I can (the syntax is killing me slowly). I've used ABCL in
>> projects before (it was wonderful, Java doesn't handle XML well).
>>
>> Lisp code:
>> (with-open-file (stream "/media/danger/OS/temp/jars.txt"
>> :external-format
>> :iso-8859-1) ; great_expectations.iso
>> (let ((size (file-length stream))
>> (buffer-size (* 16 1024 1024)) ; 16M
>> )
>> (time
>> (loop with buffer = (make-array buffer-size :element-type 'character)
>> for n-characters = (read-sequence buffer stream)
>> while (< 0 n-characters)))
>> )))
>>
>> Java code:
>> private static final int BUFFER_SIZE = 16 * 1024 * 1024;
>> try (InputStream in = new
>> FileInputStream("/media/danger/OS/temp/great_expectations.iso"); ) {
>> byte[] buff = new byte[BUFFER_SIZE];
>> int chunkLen = -1;
>> long start = System.currentTimeMillis();
>> while ((chunkLen = in.read(buff)) != -1) {
>> System.out.println("chunkLen = " + chunkLen);
>> }
>> double duration = System.currentTimeMillis() - start;
>> duration /= 1000;
>> System.out.println(String.format("it took %,2f secs", duration));
>> } catch (Exception e) {
>> e.printStackTrace(System.out);
>> } finally {
>> System.out.println("Done.");
>> }
>>
>> Robert P. Goldman
>> Research Fellow
>> Smart Information Flow Technologies (d/b/a SIFT, LLC)
>>
>> 319 N. First Ave., Suite 400
>> Minneapolis, MN 55401
>>
>> Voice: (612) 326-3934
>> Email: [email protected]
>>
Robert P. Goldman
Research Fellow
Smart Information Flow Technologies (d/b/a SIFT, LLC)
319 N. First Ave., Suite 400
Minneapolis, MN 55401
Voice: (612) 326-3934
Email: [email protected]
--=_MailMate_D7BF15E5-56CA-4574-9F3C-E92A33073DCE_=
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html>
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/xhtml; charset=3Dutf-8"=
>
</head>
<body><div style=3D"font-family: sans-serif;"><div class=3D"markdown" sty=
le=3D"white-space: normal;">
<p dir=3D"auto">I took a file of about 450MB of characters. Using SBCL, =
when I read it like this:</p>
<pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; backgr=
ound-color: #F7F7F7; border-radius: 5px 5px 5px 5px; overflow-x: auto; ma=
x-width: 90vw;"><code style=3D"margin: 0 0; border-radius: 3px; backgroun=
d-color: #F7F7F7; padding: 0px;"> (defun do-test2 ()
(with-open-file (stream *text-file*)
(let ((buffer-size (* 16 1024 1024)) ; 16M
)
(time
(loop with buffer =3D (make-array buffer-size :element-ty=
pe 'character)
for n-characters =3D (read-sequence buffer stream)
while (< 0 n-characters))))))
</code></pre>
<p dir=3D"auto">It took an average of 1.08125s to read (4 trials).</p>
<p dir=3D"auto">This procedure:</p>
<pre style=3D"margin-left: 15px; margin-right: 15px; padding: 5px; backgr=
ound-color: #F7F7F7; border-radius: 5px 5px 5px 5px; overflow-x: auto; ma=
x-width: 90vw;"><code style=3D"margin: 0 0; border-radius: 3px; backgroun=
d-color: #F7F7F7; padding: 0px;">(defun do-test3 ()
(with-open-file (stream *text-file* :element-type '(unsigned-b=
yte 8))
(let ((buffer-size (* 16 1024 1024)) ; 16M
)
(time
(loop with buffer =3D (make-array buffer-size :element-ty=
pe '(unsigned-byte 8))
for n-characters =3D (read-sequence buffer stream)
while (< 0 n-characters))))))
</code></pre>
<p dir=3D"auto">It took an average of 0.07s</p>
<p dir=3D"auto">Modifying this to set the <code style=3D"margin: 0 0; pad=
ding: 0 0.25em; border-radius: 3px; background-color: #F7F7F7;">:external=
-format</code> to <code style=3D"margin: 0 0; padding: 0 0.25em; border-r=
adius: 3px; background-color: #F7F7F7;">:iso8859-1</code> and reading int=
o an array of <code style=3D"margin: 0 0; padding: 0 0.25em; border-radiu=
s: 3px; background-color: #F7F7F7;">:element-type 'character</code> it ta=
kes an average of 0.8095s</p>
<p dir=3D"auto">So there seems to be <em>some</em> overhead to the unicod=
e handling. Note that I didn't have a file at hand that actually had ISO8=
859-1 in it, so I don't know if that would have complicated matters.</p>
<p dir=3D"auto">This suggests that just moving around bits without worryi=
ng about their interpretation <em>may</em> be faster than treating them a=
s characters. So you could see if that changes your results at all.</p>
<p dir=3D"auto">I'm not a real expert in CL file I/O, so it's likely that=
this could be done better.</p>
<p dir=3D"auto">On 21 Oct 2022, at 16:18, Garrett Dangerfield wrote:</p>
</div><blockquote class=3D"embedded" style=3D"margin: 0 0 5px; padding-le=
ft: 5px; border-left: 2px solid #777777; color: #777777;"><div id=3D"190B=
356E-24D5-45BF-B5DF-47CE3F4C5597">
<div dir=3D"ltr">
<div>I tried changing (make-array buffer-size :element-type 'character)</=
div>
<div>to</div>
<div>(make-array buffer-size :element-type 'byte)</div>
<div>and I got additional warnings and it took 70 seconds instead of 20.<=
/div>
<div><br></div>
<div>Thanks,</div>
<div>Garrett.<br></div>
</div>
<br>
<div class=3D"gmail_quote">
<div dir=3D"ltr" class=3D"gmail_attr">On Fri, Oct 21, 2022 at 1:47 PM Rob=
ert Goldman <<a href=3D"mailto:[email protected]">[email protected]<=
/a>> wrote:<br></div>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;borde=
r-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style=3D"font-family:sans-serif">
<div style=3D"white-space:normal">
<p dir=3D"auto">I don't know what data you are reading but is there any c=
hance that the difference is that when you read text in lisp as ISO-8859-=
1 lisp is actually processing the text as unicode, but when you are readi=
ng it in Java you are just slamming raw bytes into memory?</p>
<p dir=3D"auto">Maybe this is relevant? <a href=3D"https://stackoverflow.=
com/questions/979932/read-unicode-text-files-with-java" style=3D"color:rg=
b(57,131,196)" target=3D"_blank">https://stackoverflow.com/questions/9799=
32/read-unicode-text-files-with-java</a></p>
<p dir=3D"auto">I don't use Java myself, so I can't say, and I don't have=
access to your data, but it does seem like the Java code is doing someth=
ing simpler than the Lisp code.</p>
<p dir=3D"auto">What happens if you change your Lisp code to <code style=3D=
"margin:0px;padding:0px 0.25em;border-radius:3px;background-color:rgb(247=
,247,247)">read-sequence</code> of type <code style=3D"margin:0px;padding=
:0px 0.25em;border-radius:3px;background-color:rgb(247,247,247)">byte</co=
de> instead of <code style=3D"margin:0px;padding:0px 0.25em;border-radius=
:3px;background-color:rgb(247,247,247)">character</code>?</p>
<p dir=3D"auto">On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:</p>
</div>
<div style=3D"white-space:normal">
<blockquote style=3D"margin:0px 0px 5px;padding-left:5px;border-left:2px =
solid rgb(119,119,119);color:rgb(119,119,119)">
<p dir=3D"auto">I don't want to cause a firestore here but I was doing so=
me simple<br>
benchmarks on file i/o between Java, ABCL, and SBCL and I'm a bit shocked=
,<br>
honestly.</p>
<p dir=3D"auto">Reading a 2.5M file in 16M chunks in (using iso-8859-1):<=
br>
- abcl takes a tad over 1 second<br>
- sbcl takes 0.04 seconds</p>
<p dir=3D"auto">Reading a 5.8G file in 16M chunks in (using iso-8859-1 fo=
r Lisp, for Java<br>
it's just bytes):<br>
- abcl takes...too long, I gave up<br>
- sbcl takes between 20 and 21 seconds<br>
- Java takes 1.5 seconds</p>
<p dir=3D"auto">These are all run on the same computer using the same fil=
es, etc.</p>
<p dir=3D"auto">What's up with this? Thoughts? I'd heard that SBCL should=
be as fast as C<br>
under at least some circumstances. I'd wager that C is at least as fast a=
s<br>
Java (probably faster).</p>
<p dir=3D"auto">Thanks,<br>
Garrett Dangerfield. (he/him/his)</p>
<p dir=3D"auto">P.S. Don't get me wrong, I *LOVE* Lisp, I'm trying to get=
away from Java as<br>
fast as I can (the syntax is killing me slowly). I've used ABCL in<br>
projects before (it was wonderful, Java doesn't handle XML well).</p>
<p dir=3D"auto">Lisp code:<br>
(with-open-file (stream "/media/danger/OS/temp/jars.txt" :external-format=
<br>
:iso-8859-1) ; great_expectations.iso<br>
(let ((size (file-length stream))<br>
(buffer-size (* 16 1024 1024)) ; 16M<br>
)<br>
(time<br>
(loop with buffer =3D (make-array buffer-size :element-type 'character)<b=
r>
for n-characters =3D (read-sequence buffer stream)<br>
while (< 0 n-characters)))<br>
)))</p>
<p dir=3D"auto">Java code:<br>
private static final int BUFFER_SIZE =3D 16 * 1024 * 1024;<br>
try (InputStream in =3D new<br>
FileInputStream("/media/danger/OS/temp/great_expectations.iso"); ) {<br>
byte[] buff =3D new byte[BUFFER_SIZE];<br>
int chunkLen =3D -1;<br>
long start =3D System.currentTimeMillis();<br>
while ((chunkLen =3D in.read(buff)) !=3D -1) {<br>
System.out.println("chunkLen =3D " + chunkLen);<br>
}<br>
double duration =3D System.currentTimeMillis() - start;<br>
duration /=3D 1000;<br>
System.out.println(String.format("it took %,2f secs", duration));<br>
} catch (Exception e) {<br>
e.printStackTrace(System.out);<br>
} finally {<br>
System.out.println("Done.");<br>
}</p>
</blockquote>
</div>
<div style=3D"white-space:normal">
<p dir=3D"auto">Robert P. Goldman<br>
Research Fellow<br>
Smart Information Flow Technologies (d/b/a SIFT, LLC)</p>
<p dir=3D"auto">319 N. First Ave., Suite 400<br>
Minneapolis, MN 55401</p>
<p dir=3D"auto">Voice: (612) 326-3934<br>
Email: <a href=3D"mailto:[email protected]" style=3D"color:rgb(57,131,19=
6)" target=3D"_blank">[email protected]</a></p>
</div>
</div>
</div>
</blockquote>
</div></div></blockquote>
<div class=3D"markdown" style=3D"white-space: normal;">
<p dir=3D"auto">Robert P. Goldman<br>
Research Fellow<br>
Smart Information Flow Technologies (d/b/a SIFT, LLC)</p>
<p dir=3D"auto">319 N. First Ave., Suite 400<br>
Minneapolis, MN 55401</p>
<p dir=3D"auto">Voice: (612) 326-3934<br>
Email: <a href=3D"mailto:[email protected]" style=3D"color: #3983C4;"=
>[email protected]</a></p>
</div>
</div>
</body>
</html>
--=_MailMate_D7BF15E5-56CA-4574-9F3C-E92A33073DCE_=--