Re: C++ to Python and returning members of a C++ object

Can't comment on the feasibility but think having some automatic reference
counting ability would be extremely useful, even if it's only for one
language at first.

I've had to wrap an existing API that has all sorts of strange ownership
mechanics (some objects owned by the SDK, some by the application, some
methods where ownership is transferred)..We resorted to careful management
of the ownership flags and the use of an instance map in the wrappers
(Mapping to ensure each native object is only owned by one wrapped
object/reference, and ensures that returning a pointer twice from the
native api returns the same wrapped reference).

In terms of maintenance it's not too bad - Once we setup the base
capability it's just a matter of highlighting the transfers, but in terms
of efficiency it's terrible. The worst case was .Net where we end up with a
huge workload on the garbage collector trying to detangle the map of
references/etc, had to resort to lock-free containers to make it fast
enough.

Maybe it's a largely unrelated use case but if swig was able to just
reference count any C++ object that's got ownership == true it would've
saved a lot of effort and bughunting on our part (Though overall swig is
way faster than hand-generating this stuff, just forces us to test things
properly for mismatched ownership settings).

Thanks
Gareth

On Wed, 22 Jul 2020 at 10:11, Terry Barnaby <[email protected]> wrote:

> Hi,
>
> Anyone any views on this ?
> I am happy to try and create a patch to add this as a "feature" enabled
> function if people think it would be worthwhile and would be included in
> the system. It certainly seems to be a good idea from our libraries point
> of view.
>
> Terry
> On 08/07/2020 06:57, Terry Barnaby wrote:
>
> Hi William,
>
> So do you think this idea of incrementing the reference counter on the C++
> object whose member you are returning by reference is reasonable and would
> be a good addition to SWIG as a "feature" that can be enabled ? If so I can
> look at producing a proper patch for adding this (have to work out how the
> features system works first). As I don't know the SWIG code in depth it
> would need a good look at once I have done it.
>
> Terry
> On 01/07/2020 11:50, Terry Barnaby wrote:
>
> Hi William,
>
> Enclosed is my very experimental, basic and not for use example of the
> sort of thing I was thinking of.
> This is Python specific has quite a few hard coded things that won't work
> in all cases etc etc and assumes all members are returned as a pointer,
> which is incorrect.
>
> But with a lot more work, maybe something like this could be made to work
> when enabled using a particular "feature" setting ?
>
> Basically what it should do is whenever a pointer to a C++ objects member
> is returned, it will increment the reference counter on the SWIG wrapper
> for the container object and set a pointer to this container on the
> returned member pointer PyObject. Then when the member's pointer PyObject
> is deleted, it will decrement the reference counter on the overall C++
> container object so it can be deleted.
>
> I'm probably missing a few things here, but I wonder if this sort of
> mechanism could handle a lot of normal situations when pointers to C++
> members are returned ?
>
> Terry
> On 01/07/2020 06:44, Terry Barnaby wrote:
>
> Hi William,
>
> Thanks for the information. In the case of the class library I'm wrapping
> that would end up with an awful amount of code in SWIG to handle every case
> where this occurs. Most, if not all of the classes/structs I am
> encapsulating use other classes/structs as members down multiple levels and
> changes would be needed to this code for every library update. It might end
> up it would be easier not to use SWIG :)
>
> Yes, memory management, pass by value, reference/pointer etc is tough, but
> I am surprised that SWIG doesn't handle this, at least for Python, with a
> user configurable "feature" or all of the time. From what I can see for C++
> -> Python at least it would be possible to add the code, as given in the
> references you give, by default whenever a reference to a C++ object
> members data is being returned as a pointer. I think it would be a
> generally the case that if you return a reference to a C++ objects member
> you would want the overall C++ object to hang around while that pointer is
> in existence.
>
> Yes, this would reduce efficiency, but if instead of the methods suggested
> a simpler and more efficient Py_INCREF() and Py_DECREF() mechanism on the
> PyObject* to the parent object could be  used the overhead should be pretty
> small. I have tried a simple implementation of this by modifying SWIG to do
> the Py_INCREF() and it appears to work well (although it is a hack as I
> don't understand the SWIG internals well enough) and I don't know how to
> call Py_DECREF() efficiently so I am not doing this. I am assuming it would
> be possible to modify the SWIG C++ wrapper object to have the extra
> container objects pointer so its reference count can be decremented but I
> need to try and understand the SWIG code better.
>
> If you think something like this might be feasible and useful, I will have
> a look at the SWIG code a bit more in my limited spare time.
>
> Terry
> On 28/06/2020 13:32, William S Fulton wrote:
>
> Hi Terry
>
> This is one of the most horrible problem areas mixing languages, that is,
> memory management. Here is some background reading:
>
>
> http://www.swig.org/Doc4.0/Python.html#Python_memory_management_member_variables
> https://github.com/swig/swig/issues/945
> https://github.com/swig/swig/pull/1234
>
> Below is one way you can fix this (it's a variation of returning members
> by reference documented in the 1st link above). Note that SWIG uses
> pointers for accessing non-primitive member variables, see
> http://www.swig.org/Doc4.0/SWIG.html#SWIG_structure_data_members.
>
> %fragment("member_parent_attribute_init", "init") {
>   // thread safe initialization
>   member_parent_attribute();
> }
>
> %fragment("member_parent_attribute", "header",
> fragment="member_parent_attribute_init") {
>   static PyObject* member_parent_attribute() {
>     static PyObject* attr = SWIG_Python_str_FromChar("__member_parent");
>     return attr;
>   }
> }
> %typemap(ret, fragment="member_parent_attribute", noblock=1) ListA
> *TypeB::list1 %{
>   PyObject_SetAttr($result, member_parent_attribute(), $self);
> %}
>
> William
>
> On Thu, 25 Jun 2020 at 14:58, Terry Barnaby <[email protected]> wrote:
>
>> I have had a quick look at the SWIG code.
>>
>> It seems that it would be possible, and desirable to do a Py_INCREF() on
>> the PyObject* object that is wrapping the C++ object which you are
>> returning a pointer to the member from. This would hold the whole 'C++'
>> object in memory so that returning a pointer to one of this objects
>> members would be ok. The PyObject* returned for the member would have to
>> have a pointer to its parents PyObject* and do a Py_DECREF() on the
>> parent object when it is finally deleted so that the original C++ object
>> gets cleaned up eventually.
>>
>> I don't know the SWIG code to know if this is really possible and if
>> people think this is a reasonable SWIG feature that could be added ?
>> At the moment a user of your SWIG generated Python API gets a pointer to
>> a destroyed object which is not good.
>>
>> Terry
>> On 24/06/2020 10:04, Terry Barnaby wrote:
>> > I have a class library that has List objects as members of the class.
>> > If I use this class in a Python function and try to return one of
>> > these more complex members all hell breaks loose. It appears the SWIG
>> > generated wrapper returns a pointer to the objects member rather than
>> > making a copy of it when returning the member from the function.
>> >
>> > So I have something like:
>> >
>> > class TypeB {
>> >
>> > public:
>> >
>> >     List<TypeA> list1;
>> >
>> > };
>> >
>> > If in Python I have a function like the following, the returned item
>> > is effectively (SWIG wrapped) a pointer the b objects member and b
>> > disappears when the function returns.
>> >
>> > def func1():
>> >
>> >     b = TypeB();
>> >
>> >     b.list1.append(TypeA());
>> >
>> >     return b.list1;
>> >
>> > Now I can use "return ListTypeA(b.list1)", where ListTypeA has been
>> > declared using a %template declaration such that the copy constructor
>> > is called to explicitly make a copy of the list, but is there a better
>> > way of handing this ?
>> >
>> > I know this is difficult to handle, but perhaps something like a
>> > %template(return) to add code when returning a type in a Python
>> > function somehow or maybe the overall objects reference counter can be
>> > incremented ?
>> >
>> >
>> >
>> > _______________________________________________
>> > Swig-user mailing list
>> > [email protected]
>> > https://lists.sourceforge.net/lists/listinfo/swig-user
>>
>>
>>
>>
>> _______________________________________________
>> Swig-user mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>
>
>
>
>
> _______________________________________________
> Swig-user mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/swig-user
>


-- 
--------------------
Gareth Francis
www.gfrancisdev.co.uk

_______________________________________________
Swig-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/swig-user