Re: C++ to Python and returning members of a C++ object

Hi,

Anyone any views on this ?
I am happy to try and create a patch to add this as a "feature" enabled 
function if people think it would be worthwhile and would be included in 
the system. It certainly seems to be a good idea from our libraries 
point of view.

Terry
On 08/07/2020 06:57, Terry Barnaby wrote:
> Hi William,
>
> So do you think this idea of incrementing the reference counter on the 
> C++ object whose member you are returning by reference is reasonable 
> and would be a good addition to SWIG as a "feature" that can be 
> enabled ? If so I can look at producing a proper patch for adding this 
> (have to work out how the features system works first). As I don't 
> know the SWIG code in depth it would need a good look at once I have 
> done it.
>
> Terry
> On 01/07/2020 11:50, Terry Barnaby wrote:
>> Hi William,
>>
>> Enclosed is my very experimental, basic and not for use example of 
>> the sort of thing I was thinking of.
>> This is Python specific has quite a few hard coded things that won't 
>> work in all cases etc etc and assumes all members are returned as a 
>> pointer, which is incorrect.
>>
>> But with a lot more work, maybe something like this could be made to 
>> work when enabled using a particular "feature" setting ?
>>
>> Basically what it should do is whenever a pointer to a C++ objects 
>> member is returned, it will increment the reference counter on the 
>> SWIG wrapper for the container object and set a pointer to this 
>> container on the returned member pointer PyObject. Then when the 
>> member's pointer PyObject is deleted, it will decrement the reference 
>> counter on the overall C++ container object so it can be deleted.
>>
>> I'm probably missing a few things here, but I wonder if this sort of 
>> mechanism could handle a lot of normal situations when pointers to 
>> C++ members are returned ?
>>
>> Terry
>> On 01/07/2020 06:44, Terry Barnaby wrote:
>>> Hi William,
>>>
>>> Thanks for the information. In the case of the class library I'm 
>>> wrapping that would end up with an awful amount of code in SWIG to 
>>> handle every case where this occurs. Most, if not all of the 
>>> classes/structs I am encapsulating use other classes/structs as 
>>> members down multiple levels and changes would be needed to this 
>>> code for every library update. It might end up it would be easier 
>>> not to use SWIG :)
>>>
>>> Yes, memory management, pass by value, reference/pointer etc is 
>>> tough, but I am surprised that SWIG doesn't handle this, at least 
>>> for Python, with a user configurable "feature" or all of the time. 
>>> From what I can see for C++ -> Python at least it would be possible 
>>> to add the code, as given in the references you give, by default 
>>> whenever a reference to a C++ object members data is being returned 
>>> as a pointer. I think it would be a generally the case that if you 
>>> return a reference to a C++ objects member you would want the 
>>> overall C++ object to hang around while that pointer is in existence.
>>>
>>> Yes, this would reduce efficiency, but if instead of the methods 
>>> suggested a simpler and more efficient Py_INCREF() and Py_DECREF() 
>>> mechanism on the PyObject* to the parent object could be used the 
>>> overhead should be pretty small. I have tried a simple 
>>> implementation of this by modifying SWIG to do the Py_INCREF() and 
>>> it appears to work well (although it is a hack as I don't understand 
>>> the SWIG internals well enough) and I don't know how to call 
>>> Py_DECREF() efficiently so I am not doing this. I am assuming it 
>>> would be possible to modify the SWIG C++ wrapper object to have the 
>>> extra container objects pointer so its reference count can be 
>>> decremented but I need to try and understand the SWIG code better.
>>>
>>> If you think something like this might be feasible and useful, I 
>>> will have a look at the SWIG code a bit more in my limited spare time.
>>>
>>> Terry
>>> On 28/06/2020 13:32, William S Fulton wrote:
>>>> Hi Terry
>>>>
>>>> This is one of the most horrible problem areas mixing languages, 
>>>> that is, memory management. Here is some background reading:
>>>>
>>>> http://www.swig.org/Doc4.0/Python.html#Python_memory_management_member_variables
>>>> https://github.com/swig/swig/issues/945
>>>> https://github.com/swig/swig/pull/1234
>>>>
>>>> Below is one way you can fix this (it's a variation of returning 
>>>> members by reference documented in the 1st link above). Note that 
>>>> SWIG uses pointers for accessing non-primitive member variables, 
>>>> see http://www.swig.org/Doc4.0/SWIG.html#SWIG_structure_data_members.
>>>>
>>>> %fragment("member_parent_attribute_init", "init") {
>>>>   // thread safe initialization
>>>>   member_parent_attribute();
>>>> }
>>>>
>>>> %fragment("member_parent_attribute", "header", 
>>>> fragment="member_parent_attribute_init") {
>>>>   static PyObject* member_parent_attribute() {
>>>>     static PyObject* attr = 
>>>> SWIG_Python_str_FromChar("__member_parent");
>>>>     return attr;
>>>>   }
>>>> }
>>>> %typemap(ret, fragment="member_parent_attribute", noblock=1) ListA 
>>>> *TypeB::list1 %{
>>>>   PyObject_SetAttr($result, member_parent_attribute(), $self);
>>>> %}
>>>>
>>>> William
>>>>
>>>> On Thu, 25 Jun 2020 at 14:58, Terry Barnaby <[email protected] 
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>>     I have had a quick look at the SWIG code.
>>>>
>>>>     It seems that it would be possible, and desirable to do a
>>>>     Py_INCREF() on
>>>>     the PyObject* object that is wrapping the C++ object which you are
>>>>     returning a pointer to the member from. This would hold the
>>>>     whole 'C++'
>>>>     object in memory so that returning a pointer to one of this
>>>>     objects
>>>>     members would be ok. The PyObject* returned for the member
>>>>     would have to
>>>>     have a pointer to its parents PyObject* and do a Py_DECREF() on
>>>>     the
>>>>     parent object when it is finally deleted so that the original
>>>>     C++ object
>>>>     gets cleaned up eventually.
>>>>
>>>>     I don't know the SWIG code to know if this is really possible
>>>>     and if
>>>>     people think this is a reasonable SWIG feature that could be
>>>>     added ?
>>>>     At the moment a user of your SWIG generated Python API gets a
>>>>     pointer to
>>>>     a destroyed object which is not good.
>>>>
>>>>     Terry
>>>>     On 24/06/2020 10:04, Terry Barnaby wrote:
>>>>     > I have a class library that has List objects as members of
>>>>     the class.
>>>>     > If I use this class in a Python function and try to return
>>>>     one of
>>>>     > these more complex members all hell breaks loose. It appears
>>>>     the SWIG
>>>>     > generated wrapper returns a pointer to the objects member
>>>>     rather than
>>>>     > making a copy of it when returning the member from the function.
>>>>     >
>>>>     > So I have something like:
>>>>     >
>>>>     > class TypeB {
>>>>     >
>>>>     > public:
>>>>     >
>>>>     >     List<TypeA> list1;
>>>>     >
>>>>     > };
>>>>     >
>>>>     > If in Python I have a function like the following, the
>>>>     returned item
>>>>     > is effectively (SWIG wrapped) a pointer the b objects member
>>>>     and b
>>>>     > disappears when the function returns.
>>>>     >
>>>>     > def func1():
>>>>     >
>>>>     >     b = TypeB();
>>>>     >
>>>>     >     b.list1.append(TypeA());
>>>>     >
>>>>     >     return b.list1;
>>>>     >
>>>>     > Now I can use "return ListTypeA(b.list1)", where ListTypeA
>>>>     has been
>>>>     > declared using a %template declaration such that the copy
>>>>     constructor
>>>>     > is called to explicitly make a copy of the list, but is there
>>>>     a better
>>>>     > way of handing this ?
>>>>     >
>>>>     > I know this is difficult to handle, but perhaps something like a
>>>>     > %template(return) to add code when returning a type in a Python
>>>>     > function somehow or maybe the overall objects reference
>>>>     counter can be
>>>>     > incremented ?
>>>>     >
>>>>     >
>>>>     >
>>>>     > _______________________________________________
>>>>     > Swig-user mailing list
>>>>     > [email protected]
>>>>     <mailto:[email protected]>
>>>>     > https://lists.sourceforge.net/lists/listinfo/swig-user
>>>>
>>>>
>>>>
>>>>
>>>>     _______________________________________________
>>>>     Swig-user mailing list
>>>>     [email protected]
>>>>     <mailto:[email protected]>
>>>>     https://lists.sourceforge.net/lists/listinfo/swig-user
>>>>
>>>
>>
>

_______________________________________________
Swig-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/swig-user