Re: weak reference support in JyNI

Stefan,

Glad to hear of the progress on JyNI. Getting weakrefs working is going to
be essential to JyNI supporting the CPython memory model in conjunction
with Java's.

We likely need another solution, one that's closer to approach #0, but
without the poll. Basically it should be possible for using code to
register callbacks, with RefReaperThread calling each of the registered
callbacks for each GlobalRef object. A callback could enqueue (onto a
LinkedBlockingQueue for example) or do some other processing. This should
be general enough for your management purposes.

Note that the current implementation causes a recurring issue we have seen
in how threads can interact with class loaders:
http://bugs.jython.org/issue2127 This simply means we can fix this problem
in revisiting this support!

Interestingly, Jiwon Seo requested a somewhat similar notification
mechanism here, but with respect to generated bytecode,
https://bitbucket.org/jython/jython/pull-request/41/adding-bytecode-notification-mechanism
This makes sense to me - we are now trying to build more sophisticated
integrations with Jython. (Hopefully we will get that merged in soon, sorry
that has taken so long, Jiwon!)

- Jim



On Thu, May 29, 2014 at 7:17 AM, Stefan Richthofer <[email protected]
> wrote:

> Hello,
>
> I am currently working out the support of garbage collection and weak
> references in JyNI. For weak references, I would use the _weakref module
> that is already included in Jython, but I need to tweak it a bit, because
> JyNI needs to know, whether a PyCPeer (i.e. a subclass of PyObject that
> JyNI uses to wrap native objects in some cases) is weakly referenced. The
> reason for this is - roughly speaking - that in case of weak references,
> the native object should keep the peer alive (i.e. use a global ref in
> JNI), while normal references should use a peer that keeps the native
> object alive and not vise versa (i.e. use a weak global ref in JNI). So
> there will be potentially 2 versions of PyCPeer having the keep-alive
> relation
> PyCPeer ---keeps alive---> native object ---keeps alive---> PyCPeer
> Normal references should refere to the leftmost one while weak references
> should refere to the rightmost one (the python level equality function will
> report them as equal).
>
> Of course the default implementation of _weakref is not aware of this. So
> I made up roughly two (three) competing ideas how to do it.
>
> 0) The official way to get those weak references that point to a specific
> object is WeakrefModule.getweakrefs(...)
> However I would have to poll that method, which is an unprecise and
> unefficient solution. (So I don't count it as solution)
> Once a new weak reference to a PyCPeer would show up, I would replace its
> referent by the right version. Since GlobalRef extends
> java.lang.ref.WeakReference, which does not allow to modify the referent,
> this would involve creating a new GlobalRef and replacing the old one. This
> is rather tweaky since that is stored in a private field (i.e. I would use
> reflection with setAccessible(true); or let native code do the operation).
>
> 1) Use org.python.modules._weakref.GlobalRef.references to keep track.
> I would write a List implementation that wraps some other list as backend
> but reports modifications to a listener. I would use the original
> org.python.modules._weakref.GlobalRef.references as backend and insert my
> custom list into the
> org.python.modules._weakref.GlobalRef.references-field. Since it is a
> private field, this would be tweaky again, but could be done the same way
> as mentioned above. The rest would be like in "solution" 0) but the polling
> would be avoided.
>
> 2) Write an adjusted version of the _weakref-module (i.e. that checks for
> PyCPeer BEFORE creating GlobalRefs - so it works completely without
> fumbling with private fields and avoids useless recreation of GlobalRefs).
> I tried this first, because I consider it the least tweaky solution, but
> ran into problems I can't resolve. For testing and a proof of concept, I
> took the _weakref source code, put it into a new package JyNI._weakref and
> only modified the doc-string to be able to distinguish the modules.
>
> Then I adjusted the JyNI-initializer as follows:
>
> public void initialize(Properties preProperties, Properties
> postProperties, String[] argv, ClassLoader classLoader,
> ExtensiblePyObjectAdapter adapter)
> {
>                 //Customize the _weakref module to JyNI-needs:
>                 String blti = (String)
> postProperties.get("python.modules.builtin");
>                 postProperties.put("python.modules.builtin", blti == null
> ? "_weakref:JyNI._weakref.WeakrefModule" :
> blti+",_weakref:JyNI._weakref.WeakrefModule");
>
>                 //init PySystemState:
>                 PySystemState initState =
> PySystemState.doInitialize(preProperties, postProperties, argv,
> classLoader, adapter);
>
> //further initialization stuff...
>
>
> This would overwrite the _weakref-builtin with my custom version. If I do
>
> import _weakref
> print _weakref.__doc__
>
> it works nicely and indeed prints the doc-string of my custom module.
> But if I do
>
> from _weakref import ref
> print ref
> bla = "bla"
> test = ref(bla)
> print test
>
> original Jython outputs:
>
> <type 'weakref'>
> <weakref at 0x2; to 'str' at 0x3>
>
> (i.e. it works fine), while the output using my initializer is:
>
> <type 'JyNI._weakref.ReferenceType'>
> Traceback (most recent call last):
>   File
> "/home/stefan/eclipseWorkspace/JyNI/JyNI-Demo/src/JyNIWeakRefTest.py", line
> 41, in <module>
>     test = ref(bla)
> TypeError: JyNI._weakref.ReferenceType(): expected 2-3 args; got 1
>
>
> So obviously, my tweak fails to make up the right type for ref and yields
> no appropriate mro. However I have no clue how this comes, since Jython
> should initialize the custom module just the same way it would do with
> or.python.modules._weakref.WeakrefModule - it is absolutely identical apart
> from the doc-string and the package path (but that is adjusted in the
> properties).
>
> Can someone tell me why approach 2) fails and maybe how to fix it? Or
> should I work out approach 1) instead? Or do you even have a better idea
> how to reach my goal?
>
> Thanks in advance.
>
> -Stefan
>
>
> ------------------------------------------------------------------------------
> Time is money. Stop wasting it! Get your web API in 5 minutes.
> www.restlet.com/download
> http://p.sf.net/sfu/restlet
> _______________________________________________
> Jython-dev mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/jython-dev
>

------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev