Re: Jython buffer protocol

Jeff,

I just quickly scanned the changes and everything looks fine as far as I see. PyByteArray and BaseBytes may need adjustments too (on this occasion remember to add resizeCheck() in irepeat).

>>I've never really settled in my mind how I should deal with arrays of types other than byte. I keep looking at ByteBuffer.getInt() etc. and at IntBuffer, thinking those should be some sort of model.

In my opinion, the API (at least in its current design) should exclusively stick to bytes for basically two reasons: Direct buffer functionality is only available for ByteBuffers and a ByteBuffer can create an IntBuffer-, LongBuffer-, etc-view on the data. I assume this is the most efficient way that Java offers to assemble Ints, Longs etc from bytes in mass. Unfortunately, it does not offer to obtain a ByteBuffer view on an IntBuffer or LongBuffer etc (unless the starting point was ByteBuffer and one kept it). So ByteBuffer (wrapping byte[] or not) yields most functionality and should be the only data-baseline for the Buffer protocol. If a user has int[]-data, I suppose one could create a BytBuffer with an IntBuffer view and insert the data via bulk write methods.


>>Looking at how PyArray works, it almost seems that also should be based on the sub-classes of java.nio.Buffer. I think a proper, typed solution would make the itemsize property unnecessary. Do you have any thoughts about itemsize and non-byte data?

Maybe the best idea would be to use ByteBuffer and keep the itemsize to tell the user what asXXBuffer method should be used on it.



I also discussed the BufferProtocol question with Jim (rather briefly, since our main focus was gc) and he suggested it might be a good idea to introduce another indirection level between PyBuffer and ByteBuffer/byte[]. Due to inlining capabilities of the JVM, this would not have notable performance issues, but might solve some problems. For these lines I refer to the intermediate layer as "AbstractBuffer". Although ByteBuffer can be used to wrap byte[], another level would give us control about features and allow more implementations, overcoming the issue that ByteBuffer cannot be extended.
Why not directly do this with PyBuffer? It seems reasonable to have multi-dim logic above the data representation, i.e. it would make AbstractBuffer too complex.
Note that PyArray already uses such intermediate layer, i.e. AbstractArray. One would have to enhanche it to support ByteBuffers or maybe other buffers too. And of course one would have to adjust the other Built-ins and API to use this consequently.

Some ideas about what could be done with it:

- AbstractBuffer could change its backend on the fly. Maybe byte[] was appropriate at first and suddenly a direct ByteBuffer is needed. The user would still have an AbstractBuffer object, and would not even notice that backend changed. However he/she might have stored a byte[] he/she extracted from it, so there should be something in AbstractBuffer that tells the user, some view was invalidated (isValid(byte[] b) or somethng, maybe even offer an invalidation listener). For write-access there should be methods to lock and release the backend.

- AbstractBuffer could have int[] and long[] backends - one would always use the given backend directly and only convert the backend on demand, if there is a good reason. I think one would provide format info that tells the user the type of backend. If she decides to access the data in another form, this can be provided with a lack of efficiency, i.e. by converting the data view forth and back. So I imagine AbstractBuffer with a bunch of "asXXBuffer" and "asXXArray" access methods. All would be functional but not all would be efficient, depending on the actual backend. Format info tells the user which are the efficient access methods for a specific buffer.

- In future, AbstractBuffer could provide long-index access to native big data sources (maybe via a subclass LongAbstractBuffer or something)

- In general an intermediate layer under Jython's control would be the most flexible approach to currently still unknown needs and issues.


I know there would be careful thinking needed on how to design such intermediate layer and it would be a drastic change of the current API. I just wanted to make you aware of this idea. Maybe one could approach this in Jython 3 or so, or work out a minimal implementation of it, being open for more advanced use in the future.


Stefan



Gesendet: Mittwoch, 06. August 2014 um 00:42 Uhr
Von: "Jeff Allen" <[email protected]>
An: "Stefan Richthofer" <[email protected]>
Cc: "Jython Developers" <[email protected]>
Betreff: Re: Jython buffer protocol

Stefan:

I've added a method to get a ByteBuffer and one for testing the presence of array access. I've implemented this in the base classes we use to add buffers, and I've used it in core objects where previously I used PyBuffer.Pointer.

Rather than push this to the project repository immediately, I've put it here for comment:
https://bitbucket.org/tournesol/jython-ja/commits/330839dc597a4ba3cb8168e8bd859ad91b9f4dec

I think BaseBuffer needs refactoring now to separate the layer that assumes a real backing array from a base that doesn't. For now I'm minimising the changes.

I've never really settled in my mind how I should deal with arrays of types other than byte. I keep looking at ByteBuffer.getInt() etc. and at IntBuffer, thinking those should be some sort of model. Looking at how PyArray works, it almost seems that also should be based on the sub-classes of java.nio.Buffer. I think a proper, typed solution would make the itemsize property unnecessary. Do you have any thoughts about itemsize and non-byte data?
 
Jeff Allen

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls. 
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev