Re: Jython buffer protocol

>this is a breaking change to the API

I think this can be achieved without a breaking API-change (detailed comments below).
(However if you prefer a slight break to achieve a cleaner API I won't complain.)
 
The possibility that an array-storage access cannot be provided is already contained in the API.
If the flag AS_ARRAY is not set, the current API already doesn't guarantee to offer array-access (via PyBuffer.Pointer).

Does a type-change of storage field in BaseBuffer count as breaking API change, given that it is
protected? Third-parties that extend BaseBuffer might be affected, which can be avoided by
option 1), i.e. adding ByteBuffer view as a separate field, e.g. "storageBufferView".
We could start with option 1), declare the byte[]-storage field as deprecated and remove it in
2.7.4 or so. This would provide a smooth transition to variant 2).
 
Replacing PyBuffer.Pointer by ByteBuffer would be a breaking change, but could be avoided too. In Java
fashion PyBuffer.Pointer and corresponding API/methods can be kept as @deprecated. (Or just kept -
I am actually +0 about replacing PyBuffer.Pointer with ByteBuffer)


> Concerning the pointer to object member in CPython Py_Buffer, it seems to be a 3.x feature,
which may be why I haven't replicated it.
 
Taking another look it seems like this feature was actually backported to Python 2. Py_buffer declaration in
object.h of Python 2.7 is titled /* Py3k buffer interface */. However, for sake of compatibility it would
be best to support it in Jython, given that it is (presumably) easy to add.


Do I miss some aspect?

-Stefan
 

Gesendet: Mittwoch, 27. April 2016 um 23:57 Uhr
Von: "Jim Baker" <[email protected]>
An: "Jeff Allen" <[email protected]>
Cc: "Stefan Richthofer" <[email protected]>, "Jython Developers" <[email protected]>
Betreff: Re: [Jython-dev] Jython buffer protocol

On Wed, Apr 27, 2016 at 4:36 PM, Jeff Allen <[email protected]> wrote:
I'm giving serious consideration to idea 2, that is, the storage implementation is j.n.ByteBuffer, always, and *may* wrap a byte[] object. I'd need to try this out to ensure there is no fatal flaw.
*Jim:* this is a breaking change to the API. Do we need to be more careful of possible users? I suspect we are only breaking our own work here: how about you?

We should mention such a breaking change. Necessarily we have been very conservative on various aspects of our Java API - there is certainly usage out there. But that has been seen in 2.5 or earlier API definitions. I don't see a problem here - any users will be sophisticated and can readily adapt.  
 
We would be saying in this that the Jython PyBuffer is allowed to be less like the CPython one than I've been aiming for. This consistency may be less important than Stefan's use case. The CPython protocol promises efficient access to the storage of an object via a pointer, and we would be saying "only as efficient as a j.n.ByteBuffer" ... although it may turn out there's a backing array. j.n.ByteBuffer does not replace PyBuffer, because it cannot describe strided access or the get-release behaviour.
I think this leads to an API in which what I've tried to do with PyBuffer.Pointer we now do by handing out ByteBuffer slices. So Pointer goes away. In that case getBuf() and getNIOByteBuffer() are probably the same thing. I do not think it is safe to hand out the actual storage: it is almost unavoidable clients would manipulate the internal state (position, limit), surprising each other and the PyBuffer implementation if it relies on them, as I think it should.
Concerning the pointer to object member in CPython Py_Buffer, it seems to be a 3.x feature, which may be why I haven't replicated it. It seems easy to add. (I'd be rewriting all the constructors anyway.) In CPython it's null when there's a buffer but no object.
 
Jeff Allen

On 24/04/2016 15:36, Stefan Richthofer wrote:
Jeff,

good to hear that you can help with this stuff and also that your answer implies you don't have concerns with the new feature itself. Thinking it through again, I think the following way would be cleanest to add this functionality:

Add a ByteBuffer-type storage, either exclusively or in addition to byte[] storage.



1) Version with additional field java.nio.ByteBuffer bufferStorage:

Case byte[]-backed PyBuffer:
(buffer storage must be view on storage, i.e. backed by it and must always point to first element)

storage is byte[]
bufferStorage is ByteBuffer.wrap(storage)

getNIOByteBuffer() can use bufferStorage and needn't call ByteBuffer.wrap every time again.


Case direct ByteBuffer (likely not having backing array):

storage is null or if the JVM happens to be capable of providing direct ByteBuffer with byte[] backend: bufferStorage.array()

bufferStorage is ByteBuffer.allocateDirect(capacity)

Methods that used to access elements of storage directly are enriched by a fallback for case storage == null. The fallback would directly operate on bufferStorage.




2) Version with exclusive Buffer-storage:

storage type is java.nioByteBuffer instead of byte[]


Case byte[]-backed PyBuffer:

storage is ByteBuffer.allocate(capacity) (i.e. non-Direct, so buffer will have backing array!)

getNIOByteBuffer() can use storage and needn't call ByteBuffer.wrap.

Methods that used to access elements of storage directly now do this on storage.array() rather than on storage itself (should be doable by a simple search/replace refactoring more or less).


Case direct ByteBuffer (likely not having backing array):

bufferStorage is ByteBuffer.allocateDirect(capacity)

Methods that used to access elements of storage directly are enriched by a fallback for case storage.hasArray() == false. The fallback would directly operate on storage's ByteBuffer methods.


I can do the work of writing the fallbacks or help with it up to your discretion.


Then another thing: I noticed CPython's PyBuffer-pendant contains a reference to the PyObject that exported it, so you can always find the origin of a given PyBuffer. I don't see how this would be feasible with Jython's current PyBuffer implementation. So from JyNI perspective I can store (as a mapping) the exporter in case it is known for some reason, e.g. because PyBuffer was converted from a native CPython-like variant.
However there could be situations where the buffer comes from Jython and the origin would be unknown. In that case I would (currently) just provide a NULL-value or PyNone for this field and hope to get away with it for the important extensions. Maybe we could attach a PyBuffer's origin in Jython too...? (e.g. as a JyAttribute only if some global flag is set, which JyNI would then set on load).

Best

Stefan



Gesendet: Samstag, 23. April 2016 um 20:14 Uhr
Von: "Jeff Allen" <[email protected]>
An: "Stefan Richthofer" <[email protected]>
Cc: [email protected]
Betreff: Re: [Jython-dev] Jython buffer protocol

Hi Stefan.

Refreshing my memory about how these classes work, I can see that I took 
at face value the CPython view that the purpose of the buffer interface 
is to give clients access to the underlying array of bytes, so 
abstraction of the storage always gave way to what I thought would be 
efficient. (Abstraction of the unit to be something other than byte is 
sketched but clarity and a use case eluded me.)

I always feel I've failed if I have to cast. My instinct is for option a.

But I think you would not create a "Direct" parallel to BaseBuffer, 
since it contains a lot of helper methods independent of the storage 
implementation. Rather, factor it into two layers, the first being 
either BaseBuffer or AbstractBuffer (depending on what causes least 
pain) and the next layer being two base classes, one the revised 
BaseBuffer containing:
     protected byte[] storage;
and the other containing:
     protected ByteBuffer storage;
And in each you migrate case whatever it seems natural should come along 
with these declarations.

I've been meaning to get back to Jython: I could do this groundwork if 
that would not be confusing.

Jeff

Jeff Allen

On 22/04/2016 21:50, Stefan Richthofer wrote:
Hello Jeff,

I'm warming up this old thread, because I am about to start actual work on JyNI's support
for buffer-protocol / the PyBuffer builtin type.
I'd like to point you to my recent pull request https://github.com/jythontools/jython/pull/39[https://github.com/jythontools/jython/pull/39].
It's a preliminary step for adding support for direct java.nio.ByteBuffers. After establishing this flag
I am going to add some actual support for it. I see basically two ways to go for this

a) Create a parallel class hierarchy to BaseBuffer et al, backed by direct ByteBuffers. E.g.
call everything with "Direct": DirectBaseBuffer, DirectSimpleBuffer etc.
Then let BufferProtocol implementers check for the flag and use Direct counterpart of the
usually used Buffer-Class accordingly.

or

b) Modify existing BaseBuffer such that storage is Object rather than byte[]. Then according to
flags it will be byte[] or ByteBuffer. This variant will involve more explicit type casting than
a), but would involve fewer new classes however.

What is your opinion about this?

Best

Stefan


Gesendet: Donnerstag, 07. August 2014 um 21:49 Uhr
Von: "Jeff Allen" <[email protected]>
An: "Stefan Richthofer" <[email protected]>
Cc: "Jython Developers" <[email protected]>
Betreff: Re: [Jython-dev] Jython buffer protocol


Jeff Allen

On 06/08/2014 02:28, Stefan Richthofer wrote:
Jeff,

I just quickly scanned the changes and everything looks fine as far as I see. PyByteArray and BaseBytes may need adjustments too (on this occasion remember to add resizeCheck() in irepeat).
Thanks for looking that over. I was rather asking whether the use case
was served adequately by the addition of PyBUF.AS_ARRAY, hasArray() and
getNIOByteBuffer(), since you obviously have a clear idea of it.

I took a second/third look but didn't find anything to change in
BaseBytes and PyByteArray: when arguments objects have the Buffer API
they are accessed through the abstract API (not as byte[]). PyByteArray
obviously can support AS_ARRAY, and that seems to be covered in
SimpleBuffer.

I fixed the irepeat bug in the previous change set: it's a distinct
issue, so it gets its own change set. Thanks for spotting.
I know there would be careful thinking needed on how to design such intermediate layer and it would be a drastic change of the current API. I just wanted to make you aware of this idea. Maybe one could approach this in Jython 3 or so, or work out a minimal implementation of it, being open for more advanced use in the future.

I think it would make sense to have a layer below BaseBytes that
contained all those mechanisms that work without assuming a byte[]
storage. This would help you implement PyBuffer in an object unable to
export a byte[]. That wouldn't change the API and is likely harmless to
efficiency. But more radical ideas, I agree, need more careful thought.
(The present design has had a lot of thought.)

Jeff

------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk[http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk]
_______________________________________________
Jython-dev mailing [email protected]://lists.sourceforge.net/lists/listinfo/jython-dev

 
 

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev