Re: C++ to Python and returning members of a C++ object

Hi William,

Enclosed is my very experimental, basic and not for use example of the 
sort of thing I was thinking of.
This is Python specific has quite a few hard coded things that won't 
work in all cases etc etc and assumes all members are returned as a 
pointer, which is incorrect.

But with a lot more work, maybe something like this could be made to 
work when enabled using a particular "feature" setting ?

Basically what it should do is whenever a pointer to a C++ objects 
member is returned, it will increment the reference counter on the SWIG 
wrapper for the container object and set a pointer to this container on 
the returned member pointer PyObject. Then when the member's pointer 
PyObject is deleted, it will decrement the reference counter on the 
overall C++ container object so it can be deleted.

I'm probably missing a few things here, but I wonder if this sort of 
mechanism could handle a lot of normal situations when pointers to C++ 
members are returned ?

Terry
On 01/07/2020 06:44, Terry Barnaby wrote:
> Hi William,
>
> Thanks for the information. In the case of the class library I'm 
> wrapping that would end up with an awful amount of code in SWIG to 
> handle every case where this occurs. Most, if not all of the 
> classes/structs I am encapsulating use other classes/structs as 
> members down multiple levels and changes would be needed to this code 
> for every library update. It might end up it would be easier not to 
> use SWIG :)
>
> Yes, memory management, pass by value, reference/pointer etc is tough, 
> but I am surprised that SWIG doesn't handle this, at least for Python, 
> with a user configurable "feature" or all of the time. From what I can 
> see for C++ -> Python at least it would be possible to add the code, 
> as given in the references you give, by default whenever a reference 
> to a C++ object members data is being returned as a pointer. I think 
> it would be a generally the case that if you return a reference to a 
> C++ objects member you would want the overall C++ object to hang 
> around while that pointer is in existence.
>
> Yes, this would reduce efficiency, but if instead of the methods 
> suggested a simpler and more efficient Py_INCREF() and Py_DECREF() 
> mechanism on the PyObject* to the parent object could be  used the 
> overhead should be pretty small. I have tried a simple implementation 
> of this by modifying SWIG to do the Py_INCREF() and it appears to work 
> well (although it is a hack as I don't understand the SWIG internals 
> well enough) and I don't know how to call Py_DECREF() efficiently so I 
> am not doing this. I am assuming it would be possible to modify the 
> SWIG C++ wrapper object to have the extra container objects pointer so 
> its reference count can be decremented but I need to try and 
> understand the SWIG code better.
>
> If you think something like this might be feasible and useful, I will 
> have a look at the SWIG code a bit more in my limited spare time.
>
> Terry
> On 28/06/2020 13:32, William S Fulton wrote:
>> Hi Terry
>>
>> This is one of the most horrible problem areas mixing languages, that 
>> is, memory management. Here is some background reading:
>>
>> http://www.swig.org/Doc4.0/Python.html#Python_memory_management_member_variables
>> https://github.com/swig/swig/issues/945
>> https://github.com/swig/swig/pull/1234
>>
>> Below is one way you can fix this (it's a variation of returning 
>> members by reference documented in the 1st link above). Note that 
>> SWIG uses pointers for accessing non-primitive member variables, see 
>> http://www.swig.org/Doc4.0/SWIG.html#SWIG_structure_data_members.
>>
>> %fragment("member_parent_attribute_init", "init") {
>>   // thread safe initialization
>>   member_parent_attribute();
>> }
>>
>> %fragment("member_parent_attribute", "header", 
>> fragment="member_parent_attribute_init") {
>>   static PyObject* member_parent_attribute() {
>>     static PyObject* attr = SWIG_Python_str_FromChar("__member_parent");
>>     return attr;
>>   }
>> }
>> %typemap(ret, fragment="member_parent_attribute", noblock=1) ListA 
>> *TypeB::list1 %{
>>   PyObject_SetAttr($result, member_parent_attribute(), $self);
>> %}
>>
>> William
>>
>> On Thu, 25 Jun 2020 at 14:58, Terry Barnaby <[email protected] 
>> <mailto:[email protected]>> wrote:
>>
>>     I have had a quick look at the SWIG code.
>>
>>     It seems that it would be possible, and desirable to do a
>>     Py_INCREF() on
>>     the PyObject* object that is wrapping the C++ object which you are
>>     returning a pointer to the member from. This would hold the whole
>>     'C++'
>>     object in memory so that returning a pointer to one of this objects
>>     members would be ok. The PyObject* returned for the member would
>>     have to
>>     have a pointer to its parents PyObject* and do a Py_DECREF() on the
>>     parent object when it is finally deleted so that the original C++
>>     object
>>     gets cleaned up eventually.
>>
>>     I don't know the SWIG code to know if this is really possible and if
>>     people think this is a reasonable SWIG feature that could be added ?
>>     At the moment a user of your SWIG generated Python API gets a
>>     pointer to
>>     a destroyed object which is not good.
>>
>>     Terry
>>     On 24/06/2020 10:04, Terry Barnaby wrote:
>>     > I have a class library that has List objects as members of the
>>     class.
>>     > If I use this class in a Python function and try to return one of
>>     > these more complex members all hell breaks loose. It appears
>>     the SWIG
>>     > generated wrapper returns a pointer to the objects member
>>     rather than
>>     > making a copy of it when returning the member from the function.
>>     >
>>     > So I have something like:
>>     >
>>     > class TypeB {
>>     >
>>     > public:
>>     >
>>     >     List<TypeA> list1;
>>     >
>>     > };
>>     >
>>     > If in Python I have a function like the following, the returned
>>     item
>>     > is effectively (SWIG wrapped) a pointer the b objects member and b
>>     > disappears when the function returns.
>>     >
>>     > def func1():
>>     >
>>     >     b = TypeB();
>>     >
>>     >     b.list1.append(TypeA());
>>     >
>>     >     return b.list1;
>>     >
>>     > Now I can use "return ListTypeA(b.list1)", where ListTypeA has
>>     been
>>     > declared using a %template declaration such that the copy
>>     constructor
>>     > is called to explicitly make a copy of the list, but is there a
>>     better
>>     > way of handing this ?
>>     >
>>     > I know this is difficult to handle, but perhaps something like a
>>     > %template(return) to add code when returning a type in a Python
>>     > function somehow or maybe the overall objects reference counter
>>     can be
>>     > incremented ?
>>     >
>>     >
>>     >
>>     > _______________________________________________
>>     > Swig-user mailing list
>>     > [email protected]
>>     <mailto:[email protected]>
>>     > https://lists.sourceforge.net/lists/listinfo/swig-user
>>
>>
>>
>>
>>     _______________________________________________
>>     Swig-user mailing list
>>     [email protected]
>>     <mailto:[email protected]>
>>     https://lists.sourceforge.net/lists/listinfo/swig-user
>>
>

_______________________________________________
Swig-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/swig-user

diff --git a/Lib/python/pyrun.swg b/Lib/python/pyrun.swg
index 63ff82ff8..7be0bebdd 100644
--- a/Lib/python/pyrun.swg
+++ b/Lib/python/pyrun.swg
@@ -330,6 +330,7 @@ typedef struct {
 #ifdef SWIGPYTHON_BUILTIN
   PyObject *dict;
 #endif
+  PyObject *beamContainer1;
 } SwigPyObject;
 
 
@@ -740,10 +741,38 @@ SwigPyObject_New(void *ptr, swig_type_info *ty, int own)
     sobj->ty   = ty;
     sobj->own  = own;
     sobj->next = 0;
+    sobj->beamContainer1 = 0;	// Pointer to object that this object is a member of
   }
   return (PyObject *)sobj;
 }
 
+// This is rough experimental code to increase the reference count of the PyObject that wraps a C++ object when a pointer
+// to one of the C++ object's data members is returned in a PyObject. It makes sure the parent C++ object remains in
+// existance whilst a pointer to one of its members is in use.
+// The associated code in cwrap.c and python.cxx assumes all members are returned as a pointer, which is incorrect.
+SWIGRUNTIME SwigPyObject *SWIG_Python_GetSwigThis(PyObject *pyobj);
+
+// This is used when a PyObject pointing ot a C++ objects member is returned
+void BeamContainerReference(PyObject* container, PyObject* member){
+	SwigPyObject*	m = SWIG_Python_GetSwigThis(member);
+	
+	//printf("BeamContainerReference: %p %p %p\n", container, member, m);
+	Py_INCREF(container);
+	if(m)
+		m->beamContainer1 = container;
+}
+
+// This is called when a PyObject wrapping a C++ object is destroyed
+void BeamContainerDereference(PyObject* obj){
+	SwigPyObject*	cont = SWIG_Python_GetSwigThis(obj);
+	
+	printf("BeamContainerDereference: %p %p\n", obj, cont);
+	if(cont && cont->beamContainer1){
+		//printf("beamContainer1: %p\n", cont->beamContainer1);
+		Py_DECREF(cont->beamContainer1);
+	}
+}
+
 /* -----------------------------------------------------------------------------
  * Implements a simple Swig Packed type, and use it instead of string
  * ----------------------------------------------------------------------------- */
diff --git a/Source/Modules/python.cxx b/Source/Modules/python.cxx
index c8c45df35..c91e169e5 100644
--- a/Source/Modules/python.cxx
+++ b/Source/Modules/python.cxx
@@ -3143,6 +3143,10 @@ public:
       }
 #else
       Printf(f->code, "%s\n", tm);
+#endif
+#ifndef BEAM_ZAP
+      if(Getattr(n, "memberget"))
+          Printf(f->code, "BeamContainerReference(args, resultobj); // BEAM increment reference count of the container and add a prointer to the container set a pointer to this in the returned object\n");
 #endif
       Delete(tm);
     } else {
diff --git a/Source/Swig/cwrap.c b/Source/Swig/cwrap.c
index d6e5e0cdc..932fe7da1 100644
--- a/Source/Swig/cwrap.c
+++ b/Source/Swig/cwrap.c
@@ -748,6 +748,10 @@ String *Swig_cppdestructor_call(Node *n) {
   } else {
     String *pname = Swig_cparm_name(0, 0);
     String *call = NewStringf("delete %s;", pname);
+#ifndef BEAM_ZAP
+    Append(call, NewString("\nBeamContainerDereference(args); // BEAM decrement reference count of contained if needed\n"));
+#endif
+    
     Delete(pname);
     return call;
   }