Re: I wonder if I'm missing something...

I found that by using some unmentioned details about the LWM implementation, I could avoid one whole buffer copy in the operation. The high level Lisp way uses 2 whole buffer copies - Lisp to FLI, FLI back to Lisp, to effect the transformation from byte vector to float vector.

But here are details you aren’t supposed to care about:

(let ((buf (make-array 1
                       :element-type    '(unsigned-byte 8)
                       :initial-element #xAA)))
  (declare (dynamic-extent buf))
  (loenc:dump (sys:object-address buf) 40))

____   0  1  2  3   4  5  6  7   8  9  A  B   C  D  E  F
0000  36 2B 00 00  08 00 00 00  AA AA AA AA  AA AA AA AA  6+......ªªªªªªªª
0010  36 2B 00 00  40 01 00 00  55 55 55 55  55 55 55 55  [email protected]
0020  55 55 55 55  55 55 55 55                            UUUUUUUU

Byte #4 shows the length of the array, here just 1 element. There must be 3 tag bits on Fixnums. But the array arena has 8 of them, all initialized to my initial element #xAA, starting at offset 8 from the object address.

So the interesting thing is that you can’t allocate anything but chunks of memory 8 bytes long. And even though my array has only one element, the system allocated 8 of them and initialized them too.

But the takeaway is that a simple raw array has an 8 byte prefix at the address given by SYS:OBJECT-ADDRESS. And by declaring the buffer to have DYNAMIC-EXTENT, we force the array to live on the system stack, and not move due to GC (unless the stack gets relocated….)

So now, I can read :UINT8 bytes from a Socket stream, directly into a stack allocated octet buffer, then turn around and use the FLI:REPLACE-FOREIGN-ARRAY to convert that buffer of octets into a buffer of single-float values in just one copy operation:

	(let ((buf   (make-array  16
                                  :element-type ‘(unsigned-byte 8))))
	   (declare (dynamic-extent buf))
	   (READ-SEQUENCE buf socket-stream)
	   (FLI:REPLACE-FOREIGN-ARRAY my-float-vec (fli:make-pointer :type :FLOAT
                                                                     :address (+ 8 (SYS:OBJECT-ADDRESS buf)))
                                      :end2 4))



> On Nov 8, 2025, at 11:13, David McClain <[email protected]> wrote:
> 
> Okay,
> 
> I built my own wrapper called ADAPTER-STREAM around the network Socket stream. And I wrote my own STREAM:STREAM-READ-SEQUENCE specialized to the ADAPTER-STREAM.
> 
> It works. 
> 
> But it still moves data that really shouldn’t need to be moved. I just moved the Simon FLI code to the inside of the new STREAM:STREAM-READ-SEQUENCE code.
> 
> From high level LW what is needed is for SYS:TYPED-AREF vectors to be read into by unsigned byte streams. That can’t happen because the element type of these special vectors is DOUBLE-FLOAT. And you can’t substitute your own ‘(UNSIGNED-BYTE 8) vector for use in SYS:TYPED-AREF - it knows you are trying to cheat.
> 
> And even if we could pull this off in LW, there is probably no way to port that over to SBCL. But that is of secondary concern to me.
> 
> 
>> On Nov 8, 2025, at 10:28, David McClain (as dbm at refined-audiometrics dot com) <[email protected]> wrote:
>> 
>> I just tried building my own READ-SEQUENCE in LW.  You can’t really do that because READ-SEQUENCE is a DEFUN.
>> 
>> But you can define your own Method for STREAM:STREAM-READ-SEQUENCE. And that does what I need.
>> 
>> 
>> 
>>> On Nov 8, 2025, at 10:21, Marco Antoniotti (as marco dot antoniotti at unimib dot it) <[email protected]> wrote:
>>> 
>>> 
>>> 
>>> On Sat, Nov 8, 2025 at 6:15 PM David McClain <[email protected] <mailto:[email protected]>> wrote:
>>>> Meanwhile, I see that it is now possible to implement the ideas that you stated about Stream Adapters. 
>>>> 
>>>> I thought you were just musing about extensions that you would like to see added to Lisp.  I didn’t realize that we could implement our own READ-SEQUENCE until now. 
>>> 
>>> You can with SBCL because most sequence functions are somewhat generic (I do not have much experience using them).  Not sure about LW, or the ANSI.
>>> 
>>> I am curious about how you'd do it in LW (even portably to, say, ECL)
>>> 
>>> Cheers
>>> 
>>> Marco
>>> 
>>> 
>> 
>