GC, weakref and finalization transparency and compatibility project

Hello everybody,

I am currently working to get better transparency and compatibility into

Jython's garbage collection, finalization and weak reference handling

processes (details below). This prepares gc support in JyNI and its

debugging, which is my main motivation. However, this project will also

fix some bugs, allow for better debugging of memory leaks (yes, also Java

can have memory leaks - e.g. if one forgets to clear caches or to use weak

references in the right places), will allow for better comparision to

CPython behavior, allow to emulate non-async gc-run and provide a

more complete implementation of the gc module.

It divides -more or less naturally- into 4 steps:

1) Finalization (already done and merged in)

Reformation of Jython's finalization process. Now finalizers are supported

for new-style classes (c.f. issue 1057). Read more on this in the java doc

of org.python.core.finalization.FinalizablePyObject.java.

2) General purpose Attributes (already done, but not merged)

See issue 2224. This fixes the breakage of id() in resurrection case and

prepares to add feature to optionally preserve weak references in

resurrection case (planned for step 4).

It will also allow JyNI to link native objects to Java PyObjects in a

resurrection-save way.

(Does anybody know another usecase in Jython where a weak hash map is

used to attach some attributes to arbitrary PyObjects except Py.id()

and weak references?)

3) Traverseproc (already done, but not merged)

Implement a Jython-equivalent of CPython's Traverseproc mechanism. This

allows to iterate through the PyObjects that are referenced by a given

PyObject. It enables more transparency of the gc-process, allows to

implement some more methods of the gc-module properly and generally allows

better gc-related debugging. It eases the task of finding memory leaks and

prepares basic functionality to debug JyNI's upcoming gc emulation.

I implemented Traverseproc for most prominent builtins, but would need

some hints, where else it should be implemented. So far I implemented it

for:

BaseSet, PyCell, PyClass, PyDictionary, PyFrame, PyFunction,

PyInstance, PyMethod, PyModule, PySequenceList, PyStringMap,

PyType and all fooDerived classes (i.e. in the generator).

4) Compatibility mode and enhancing the gc-module (partly done)

This step establishes some optional compatibility flags for gc and

finalization that bring Jython's behavior closer to CPython's behavior.

As this comes with some cost, this functionality is optional and can be

turned on and off with some flags. This step will change the attitude

regarding to subtle gc related incompatibilities. It puts the Jython user

into control of compatibility versus efficiency rather than the JVM's

natural restrictions.

- gc.collect() will optionally block until gc run and all side effects of

finalizers are processed (like it does in CPython)

- gc.collect() will optionally report how many objects were collected

- gc.collect() will optionally report how many objects would have been

collected by CPython in the same situation (i.e. count only those

participating in reference cycles)

- if the right flag is active, the majority of tests in test_gc will

presumably pass

- validity of weak references and the result of weakref.getweakrefs(obj)

can optionally be preserved in resurrection case

- CPython < 3.4 does not run finalizers that are in a ref cycle. Instead

it moves corresponding objects to gc.garbage for manual cleanup. This

behavior can optionally be emulated by Jython.

- allows JyNI to handle some "edgy cases"

As some of these steps trigger significant changesets to Jython, I assume

there is maybe need for discussion and I am open to feedback.

You don't need to expect much additional work-load as I have already

done and tested most of the work to ensure feasibility. Some reviewing

efforts cannot be avoided though. I took care that additional performance-

and memory-consumption is minimal and in fact not notable as long as

all flags introduced in 4) are turned off. All new functionality is

confirmed via unit tests and I took care to avoid any regressions

according to regrtests. Finally I would like to get this functionality

into Jython 2.7.0.

Regards

-Stefan

------------------------------------------------------------------------------

_______________________________________________
Jython-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jython-dev