Re: Buffer protocol - direct vs. JVM ByteBuffers

On 17/05/2016 01:52, Stefan Richthofer wrote:
>
> The thread could create two (or more) PyBuffer-views of the same 
> object and hand both to various functions that read and write on them 
> without calling release (and thus trigger copy-back) inbetween. The 
> extension would expect if view 'A' was modified, view 'B' already 
> reflects this modification when passed to another function.
The object can hand out a second reference to the same PyBuffer. (It's 
not required to, but the built-ins do.)

> > The only way I can imagine an object with Java fields as storage 
> giving you a direct ByteBuffer on demand is to allocate one and copy 
> its state there. [...] effectively a change of implementation on the fly.
> This is what I have in mind. Changing the backend on the fly shouldn't 
> be much more expensive than creating a copy, but would then save the 
> cost of copy-back and entire copy cost for future calls. Using the 
> bulk-set method of ByteBuffer this should be easy and efficient to do 
> (bulk-get to convert the other direction). The only infeasible 
> situation would be if an AS_DIRECT_NIO buffer was requested while an 
> array-backed buffer is exported and not yet released or vise versa.
Aye, there's the rub, if at least one is writable.

> In this case the request should just fail. I suppose for sake of 
> debugging we should add a verbose mode/flag that makes Jython print 
> out (or append it to the error message in bufferErrorFromSyndrome) the 
> exact reason why some buffer-request failed so a user is able to 
> identify the design flaw.
Exceptions should always be that clear. However, it's not really a 
design flaw: hold a memoryview, and call a numpy function on the array: 
it's hardly faulty logic.
> > This seems to bring us full circle to the behaviour of 
> Get<PrimitiveType>ArrayElements, except that I suppose the object 
> knows it has done it and can handle intervening access via the Java API
> Since actual views to the same memory would be shared with (native) 
> extensions unlike Get<PrimitiveType>ArrayElements (in copy-case/no 
> array pinning) this would not break the case described above, where 
> multiple PyBuffer views to the same object are in the game.
> I'm not sure what you mean by "full circle to the behaviour of 
> Get<PrimitiveType>ArrayElements",
I meant in the sense that we have made a copy especially for C and may 
have to copy it back. You're correct that there are still a number of 
delicate problems to solve during the period the implementation has changed.

Jeff
> *Gesendet:* Montag, 16. Mai 2016 um 22:04 Uhr
> *Von:* "Jeff Allen" <[email protected]>
> *An:* "Stefan Richthofer" <[email protected]>
> *Cc:* "Jython Developers" <[email protected]>
> *Betreff:* Re: [Jython-dev] Buffer protocol - direct vs. JVM ByteBuffers
>
> Thanks for that. So, the (possible) copy-back semantics of 
> Get<PrimitiveType>ArrayElements effectively make a direct buffer copy 
> of the contents. If the C-code runs in the thread of the Java 
> execution that calls it (or equivalently it is suspended) then to 
> first order the copy causes no problem. But it is not difficult to 
> cook up a scenario where another thread or call-back into Java sees a 
> different state from C.
>
> Alternatively, one uses the "critical" methods, and suffers 
> restrictions that are, I expect, unenforcible on arbitrary CPython 
> extension modules, such as being short and not yielding the CPU.
>
> I'm reminded of the relationship in CPython between C-code and 
> interpreted code, where the GIL must be held, proving all other 
> threads are "restubg" between instructions, and a context switch is 
> only allowed when surrounded by the appropriate magical incantations. 
> I think the Universe is trying to tell us something.
>
> The problem I see with the DIRECT_NIO flag is that one cannot expect 
> to choose, at the point of getting a PyBuffer, whether that buffer 
> should be direct or heap. The data that hold the state of an object 
> have a certain implementation in Java, and so the buffer will be a 
> heap buffer. Or one can imagine a PyObject whose state is always in a 
> direct ByteBuffer (representing an image mapped from disk, say) and 
> then the PyBuffer would always be direct. Just possibly objects whose 
> main purpose is to be native-friendly would have that implementation. 
> Just possibly, this is a thing you get to choose when the object is 
> constructed.
>
> The only way I can imagine an object with Java fields as storage 
> giving you a direct ByteBuffer on demand is to allocate one and copy 
> its state there. This seems to bring us full circle to the behaviour 
> of Get<PrimitiveType>ArrayElements , except that I suppose the object 
> knows it has done it and can handle intervening access via the Java 
> API ... effectively a change of implementation on the fly.
>
> Jeff Allen
> On 15/05/2016 16:39, Stefan Richthofer wrote:
>
>     Just an add-on to my recent post:
>     >It may depend on storage.hasArray(), but storage.isDirect() seems
>     to make no difference.
>     I think it is likely the JVM does not offer a backing array, if
>     the buffer is created as direct (i.e. these flags likely exclude
>     each other), because this would imply array pinning and all the
>     restrictions coming with it. I didn't test it though, but anyway
>     we cannot rely on the one or other behavior, as doc explicitly
>     does not guarantee a backing array for direct buffers, saying this
>     is "implementation specific".
>     *Gesendet:* Sonntag, 15. Mai 2016 um 11:57 Uhr
>     *Von:* "Jeff Allen" <[email protected]>
>     *An:* "Stefan Richthofer" <[email protected]>, "Jython
>     Developers" <[email protected]>
>     *Betreff:* [Jython-dev] Buffer protocol - direct vs. JVM ByteBuffers
>
>     Stefan:
>
>     https://github.com/jythontools/jython/pull/39
>
>     What difference does it make in your use case whether a NIO
>     ByteBuffer is direct or non-direct? I can see why a client might
>     want to know which it had been given, but not why it might want an
>     exception raised in one or other case.
>
>     Nothing I'm doing seems to depend on what kind of memory the
>     exporting object has, therefore on the implementation type of
>     ByteBuffer storage. It may depend on storage.hasArray(), but
>     storage.isDirect() seems to make no difference.
>
>     Jeff
>
>     -- 
>     Jeff Allen
>
>

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev