Re: File I/O Metrics

--000000000000f2519505ec5eb52f
Content-Type: text/plain; charset="UTF-8"

Here's an example of fast reading in ABCL. The implementation of certain
array types use java.nio buffers, which can be directly read into via the
java.nio functions.

(defun test-read (path)
  (let* ((f (new 'RandomAccessFile (namestring (truename path)) "r"))
         (channel (#"getChannel" f))
         (array (make-array (* 16 1024 1024) :element-type '(unsigned-byte
8)))
         (buffer (get-java-field array "elements" t)))
    (time (loop for count = (#"read" channel buffer)
                until (eql count -1)
                sum count
                do (#"position" buffer 0)
                ))))

On my machine, for a 5G file, the SBCL code in an earlier post takes 2.4
seconds. This code takes 1.4 sec. It's fastest if I use 2M buffers - 1.1
seconds. SBCL is also marginally faster with smaller buffer sizes.

In the ABCL source code the files "SimpleArray_*.java" are the
implementations of the nio buffer backed array types. See make_array.java
where the specific type of underlying array is chosen. There is a global
switch for array allocation choosing either direct allocation or
nio-buffers, with the default being nio buffers.

(get-java-field 'java$buffers "active" t)
-> #<org.armedbear.lisp.Java$Buffers$AllocationPolicy NIO {432E958E}>

I haven't looked into the :element-type 'character case.

Maybe someone who is familiar with the ABCL stream implementation is
interested in writing a fast path for read-sequence that uses the nio
calls? If so, shout. Otherwise I'll keep it on my
procrastinate-by-hacking-abcl list.

Alan

--000000000000f2519505ec5eb52f
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Here&#39;s an example of fast reading in ABCL. The im=
plementation of certain array types use java.nio buffers, which can be dire=
ctly read into via the java.nio functions.</div><div><br></div><div><span s=
tyle=3D"font-family:monospace">(defun test-read (path)</span></div><span st=
yle=3D"font-family:monospace">=C2=A0 (let* ((f (new &#39;RandomAccessFile (=
namestring (truename path)) &quot;r&quot;))<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0(channel (#&quot;getChannel&quot; f))<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0(array (make-array (* 16 1024 1024) :element-type &#39;(unsigned-byte=
 8)))<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(buffer (get-java-field array &q=
uot;elements&quot; t)))<br>=C2=A0 =C2=A0 (time (loop for count =3D (#&quot;=
read&quot; channel buffer)<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 until (eql count -1)<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 sum count<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 do (#&quot;position&quot; buffer 0)<br></span><div><span styl=
e=3D"font-family:monospace">=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 ))))</span></div><div><br></div><div>On my machine, for a 5G fil=
e, the SBCL code in an earlier post takes 2.4 seconds. This code takes 1.4 =
sec. It&#39;s fastest if I use 2M buffers - 1.1 seconds. SBCL is also margi=
nally faster with smaller buffer sizes.<br></div><div><br></div><div>In the=
 ABCL source code the files &quot;SimpleArray_*.java&quot; are the implemen=
tations of the nio buffer backed array types. See make_array.java where the=
 specific type of underlying array is chosen. There is a global switch for =
array allocation choosing either direct allocation or nio-buffers, with the=
 default being nio buffers.</div><div><br></div><div>(get-java-field &#39;j=
ava$buffers &quot;active&quot; t)</div><div>-&gt; #&lt;org.armedbear.lisp.J=
ava$Buffers$AllocationPolicy NIO {432E958E}&gt;=C2=A0</div><div><br></div><=
div>I haven&#39;t looked into the :element-type &#39;character case. <br></=
div><div><br></div><div>Maybe someone who is familiar with the ABCL stream =
implementation is interested in writing a fast path for read-sequence that =
uses the nio calls? If so, shout. Otherwise I&#39;ll keep it on my procrast=
inate-by-hacking-abcl list.<br></div><div><br></div><div>Alan<br></div><div=
><br></div></div>

--000000000000f2519505ec5eb52f--