Re: Jython buffer protocol

I'm editing for size in the hope it makes it to the list this time

On 18/07/2014 15:01, Stefan Richthofer wrote:
> Dear Jeff,
> - Your suggestion for #1 with the new PyBUF-flag seems fine for me. My 
> only worry would be that in most situations PyBuffers would support 
> PyBuffer.Pointer even if requested without the flag. This itself is 
> okay, but since programmers are not familiar with the new flag, they 
> might get used to get away with not using it. Maybe with a clear 
> warning in the doc this would go fine.
Yes, the flags are confusing, even for me. I take the point that we're 
now asking the client to understand another. I see it working in the 
same sense as READONLY: the client signals that it will *not* be using a 
feature the exporter *might not* be willing to provide. Maybe it is 
simpler if the implementation simply throws when a Pointer is requested.
> - I was not aware that package-private stuff prevents one from 
> extending ByteBuffer. Stack-Overflow says this allows the JVM to do 
> significant optimization, but indeed ByteBuffers appear less useful to 
> me than before.
You have to call super() explicitly, as there is no default constructor, 
but the declared ones are not visible.
> - Your suggestion for #1 made me recall that also CPython's 
> BufferProtocol takes into account that an exporter might raise an 
> error if it can't deal with the requested PyBUF-flags. I can use this 
> behaviour in #2 to deal with the case that JNI can't provide direct 
> access to an array's elements. However the downside of this "solution" 
> would be inconsistent behavior between Jython's and JyNI's 
> BufferProtocols - PyBUF flags that work on Jython might fail on native 
> side.
The flags don't mean quite the same things anyway. Jython's mean "I can 
cope with", where CPython's mean "I can cope with and am going to use". 
The difference is that clients using only the abstracted API can cope 
with any buffer organisation, since they don't actually use (for 
example) the strides array.
> - I agree on your concerns for #2 regarding efficiency (although I can 
> imagine, that ByteBuffers would still perform rather well as far as 
> one sticks to bulk get/put-operations; ByteBuffers are said to be well 
> optimized by the JVM). However, I had an implementation in mind that 
> distinguishes direct ByteBuffer vs byte[]-backed buffer case early, 
> i.e. at Jython startup (via the proposed config parameter). Jython 
> could choose the appropriate implementations of 
> BufferProtocol-supporting builtin types and assign them to the 
> builtins in the builtin-type-dictionary. I admit this is a bunch of 
> work, but it is not so urgent and I would offer to provide the 
> alternative implementations or at least help with it. However I would 
> postpone this until an actual use-case comes up (i.e. an extension 
> appears that could not be made working cleanly without it). With your 
> suggestion for #1/new PyBUF-flag, Jython's BufferProtocol would be 
> flexible enough to insert such enhanchments later.
The JUnit tests on PyByteArray (or is it BaseBytes?) also time 
insertion, appending and deleting, which would be helpful regarding 
performance achieved.

Do we know how CPython would deal with an object claiming to support the 
buffer protocol, but that couldn't furnish a char* buf member?
> - One more thing. I recently recognized that the method "irepeat" in 
> PyByteArray (used by bytearray___imul__) replaces the storage backend 
> by a new array of changed size. But it does not call resizeCheck(). Is 
> this a bug, or do I miss something? (I admit I am currently not 
> looking at the newest beta-code, so please ignore this if it was 
> already fixed.)
Good spot. That's a bug:
 >>> b = bytearray('hello')
 >>> m = memoryview(b)
 >>> b.append(' ')
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
BufferError: Existing exports of data: object cannot be re-sized
 >>> b*=3       # should raise the same error
 >>> b
bytearray(b'hellohellohello')
 >>> bytearray(m)
bytearray(b'hello')

> -Stefan
> *Gesendet:* Donnerstag, 17. Juli 2014 um 22:18 Uhr
> *Von:* "Jeff Allen" <[email protected]>
> *An:* "Stefan Richthofer" <[email protected]>
> *Cc:* "Jython Developers" <[email protected]>, "Jim 
> Baker" <[email protected]>
> *Betreff:* Re: Jython buffer protocol
> Dear Stefan:
>
> I did indeed mean move it to jython-dev, not the other place. Thanks 
> for the careful exposition of your ideas.
>
> In pursuit of idea #1, I think it would not be too difficult to 
> present a ByteBuffer as a PyBuffer, even when there is no byte[] 
> behind it. There are obvious implementations for most of the 
> "abstract" API, and I imagine getBufferSlice could be done with 
> ByteBuffer.slice().
>
> We could add a getNIOByteBuffer to PyBuffer with a default 
> implementation to wrap a Pointer. That is the most significant use of 
> Pointer in the core already. I think I even added it once and reverted 
> it as gold-plating. I see a reason for it now: an object not 
> presenting byte[] access, could implement that method its own way.
>
> I am not keen to lose the direct access to the underlying byte[] that 
> Pointer gives, in general. It seems to me that this is the essence of 
> the CPython Py_Buffer: it gives you a char* and some dimensions, and 
> it lets you have at the data directly, in whatever pattern of access 
> you need, and quickly. A PyBuffer.Pointer is the Java equivalent of 
> the char *buf member. However, I think it would be acceptable for 
> certain objects not to implement the operations that return a Pointer, 
> or (like PyString) to implement them expensively, but defer the cost. 
> I appreciate that implementation may be practical only for a read-only 
> object.
>
> We could add a PyBUF-flag, whereby a client says whether it expects to 
> make Pointer access, so getBuffer(int) could fail early and helpfully, 
> rather than the client fail later.
>
> Idea #2 sounds very difficult to pull off at all (at the JyNI end). 
> And to make it possible you imply that all core objects that implement 
> BufferProtocol would be rewritten to store their data in a ByteBuffer. 
> BaseBytes has a lot in common with ByteBuffer, it is true, but the 
> difference is that inside BaseBytes I can get at the bytes directly, 
> and the clients can do so through a PyBuffer.Pointer. By comparison, 
> using ByteBuffer feels like working through a keyhole. It cannot even 
> be extended, since its constructor is package-private. It is possible 
> to do this, I would say, but to forbid Jython efficient access to data 
> as byte[], in order to afford C-code direct access to it as char*, 
> seems perverse.
> Jeff Allen
> On 17/07/2014 13:40, Stefan Richthofer wrote:
>
>     Dear Jeff,
>     the true goal I want to achieve is a clean BufferProtocol support
>     in JyNI. That means two things in my opinion.
>     1) If a CPython extension features types that support
>     BufferProtocol and these are passed to Jython via JyNI, they shall
>     appear as Jython PyObjects that support Jython's BufferProtocol.
>     The exact memory-data that the CPython object exposes shall be
>     exposed in Java by the Jython object. Optimally this should work
>     for reading and writing and as direct as possible (for efficiency,
>     minimizing memory requirements and -most important- to guarantee
>     sync between Java-view and native view on the data). Using direct
>     ByteBuffers this could be achieved, at least for JVMs that support
>     these. On other JVMs, I would use byte[] as a fallback and try to
>     obtain native access on the memory. However in this case, JNI does
>     not guarantee to provide access to the actual memory of the array.
>     It might only offer a copy, which would hold the risk of loosing
>     sync. I see some techniques how to avoid that somehow, but it
>     won't be very efficient nor elegant, nor absolutely save.
>     2) If a Jython object implements Jython's BufferProtocol in Java
>     and JyNI is used to pass a PyObject from Jython down to a CPython
>     extension, the native variant of the object shall support
>     CPython's BufferProtocol. The object shall expose the
>     corresponding memory from the JVM to the extension via this
>     protocol. This shall work for reading and writing and as direct as
>     possible for same reasons as above. I admit, this will be hard for
>     Jython objects where a user implements PyBuffer in his own
>     fashion. However, I think at least Jython's built-in types could
>     and should support this. It can also be done with direct ByteBuffers.
>     I know that direct ByteBuffers also have their disadvantages. They
>     prohibit the JVM from doing memory optimization. Additionally the
>     GC does not consider their memory when it determines what to
>     delete or when to run (it would still clear the memory, if it
>     collects a buffer). So direct ByteBuffers should only be used when
>     really needed. I would propose a configuration-parameter for
>     Jython that tells BaseBytes and BaseBuffer to use direct
>     ByteBuffers. If this parameter is turned off, they could use
>     ByteBuffer.wrap to fall back to the current implementation.
>     The doc of Jython's BufferProtocol could tell potential other
>     implementers also to look at this parameter and store their data
>     as direct or ordinary ByteBuffer accordingly. However, I would
>     build fallbacks into JyNI to deal as good as possible with
>     situations where implementers don't stick to this.
>     Both scenarios would mean that the backend in BaseBytes and
>     BaseBuffer had to be of a type that unifies byte[] and ByteBuffer.
>     One variant would be to use ByteBuffer as type and ByteBuffer.wrap
>     for byte[] case. Another variant would be to have the storage of
>     type Object. The PyBuffer implementation would know whether it is
>     byte[] or ByteBuffer, or maybe even String and work accordingly
>     (this would involve lots of explicit type casts though). However
>     PyBuffer.pointer should not provide a byte[] and pretend it to be
>     the actual backend in either variant.
>     I think the solution from PyString, i.e. SimpleStringBuffer only
>     works well for read-only scenarios, since it would lead to
>     asynchronity between byte[] and String backends if the user uses
>     the obtained byte[] for writing (assuming a mutable String variant
>     like StringBuffer). Additionally, such an approach potentially
>     doubles the memory requirements, which might be significant in
>     some situations.
>     I mentioned my thoughts about support for (>2^31 bytes)-arrays,
>     because -afaIk- a CPython extension might expose such long data
>     via BufferProtocol and I was wondering how to deal with this. Then
>     I mentioned it as a maybe misleading example. I could imagine to
>     provide kind of LongPyBuffer that enhanches PyBuffer by long-index
>     methods and allows to obtain ordinary int-sized slices to portions
>     of the data. But these are future thoughts - for a first step I
>     would just document it as a JyNI-limitation to support
>     BufferProtocol only for (>2^31 bytes)-buffers.
>
>     Again - I would be fine with postponing this decision if the
>     public API was adjusted to be open for more variants than byte[].
>     I would also offer to help with implementing the adjustments and
>     -if accepted- the refining of Jython's BufferProtocol.
>     I hope this explains my intentions a bit better.
>
>     Cheers
>     Stefan
>


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds