Re: C++ to Python and returning members of a C++ object
Terry Barnaby <[email protected]>
| Newsgroups | gmane.comp.programming.swig |
|---|---|
| Message-ID | <[email protected]> |
Yes, this won't handle all of the various nasties of memory allocation
and timing that can occur with some libraries, but at least it will
handle returning/using a C++'s struct/class data member in, I think, the
vast majority of cases. And its a very simple addition to SWIG for
Python and quite efficient, unless I have missed something. Our library
has structures and classes that have other structs/classes as data
members like (very simplified idea):
class Info {
public:
BString name;
ListofAges ages;
};
If in Python, with a standard SWIG wrapped C++ library, you do:
def getAges():
info = getInfoCFunc();
return info.ages;
All hell breaks loose (info disappears and thus the return info.ages
ends up being a pointer to a deleted class's member), but with the
simple reference counting feature, this works fine.
Terry
On 22/07/2020 12:04, Gareth Francis wrote:
> Can't comment on the feasibility but think having some automatic
> reference counting ability would be extremely useful, even if it's
> only for one language at first.
>
> I've had to wrap an existing API that has all sorts of strange
> ownership mechanics (some objects owned by the SDK, some by the
> application, some methods where ownership is transferred)..We resorted
> to careful management of the ownership flags and the use of an
> instance map in the wrappers (Mapping to ensure each native object is
> only owned by one wrapped object/reference, and ensures that returning
> a pointer twice from the native api returns the same wrapped reference).
>
> In terms of maintenance it's not too bad - Once we setup the base
> capability it's just a matter of highlighting the transfers, but in
> terms of efficiency it's terrible. The worst case was .Net where we
> end up with a huge workload on the garbage collector trying to
> detangle the map of references/etc, had to resort to lock-free
> containers to make it fast enough.
>
> Maybe it's a largely unrelated use case but if swig was able to just
> reference count any C++ object that's got ownership == true it
> would've saved a lot of effort and bughunting on our part (Though
> overall swig is way faster than hand-generating this stuff, just
> forces us to test things properly for mismatched ownership settings).
>
> Thanks
> Gareth
>
> On Wed, 22 Jul 2020 at 10:11, Terry Barnaby <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi,
>
> Anyone any views on this ?
> I am happy to try and create a patch to add this as a "feature"
> enabled function if people think it would be worthwhile and would
> be included in the system. It certainly seems to be a good idea
> from our libraries point of view.
>
> Terry
> On 08/07/2020 06:57, Terry Barnaby wrote:
>> Hi William,
>>
>> So do you think this idea of incrementing the reference counter
>> on the C++ object whose member you are returning by reference is
>> reasonable and would be a good addition to SWIG as a "feature"
>> that can be enabled ? If so I can look at producing a proper
>> patch for adding this (have to work out how the features system
>> works first). As I don't know the SWIG code in depth it would
>> need a good look at once I have done it.
>>
>> Terry
>> On 01/07/2020 11:50, Terry Barnaby wrote:
>>> Hi William,
>>>
>>> Enclosed is my very experimental, basic and not for use example
>>> of the sort of thing I was thinking of.
>>> This is Python specific has quite a few hard coded things that
>>> won't work in all cases etc etc and assumes all members are
>>> returned as a pointer, which is incorrect.
>>>
>>> But with a lot more work, maybe something like this could be
>>> made to work when enabled using a particular "feature" setting ?
>>>
>>> Basically what it should do is whenever a pointer to a C++
>>> objects member is returned, it will increment the reference
>>> counter on the SWIG wrapper for the container object and set a
>>> pointer to this container on the returned member pointer
>>> PyObject. Then when the member's pointer PyObject is deleted, it
>>> will decrement the reference counter on the overall C++
>>> container object so it can be deleted.
>>>
>>> I'm probably missing a few things here, but I wonder if this
>>> sort of mechanism could handle a lot of normal situations when
>>> pointers to C++ members are returned ?
>>>
>>> Terry
>>> On 01/07/2020 06:44, Terry Barnaby wrote:
>>>> Hi William,
>>>>
>>>> Thanks for the information. In the case of the class library
>>>> I'm wrapping that would end up with an awful amount of code in
>>>> SWIG to handle every case where this occurs. Most, if not all
>>>> of the classes/structs I am encapsulating use other
>>>> classes/structs as members down multiple levels and changes
>>>> would be needed to this code for every library update. It might
>>>> end up it would be easier not to use SWIG :)
>>>>
>>>> Yes, memory management, pass by value, reference/pointer etc is
>>>> tough, but I am surprised that SWIG doesn't handle this, at
>>>> least for Python, with a user configurable "feature" or all of
>>>> the time. From what I can see for C++ -> Python at least it
>>>> would be possible to add the code, as given in the references
>>>> you give, by default whenever a reference to a C++ object
>>>> members data is being returned as a pointer. I think it would
>>>> be a generally the case that if you return a reference to a C++
>>>> objects member you would want the overall C++ object to hang
>>>> around while that pointer is in existence.
>>>>
>>>> Yes, this would reduce efficiency, but if instead of the
>>>> methods suggested a simpler and more efficient Py_INCREF() and
>>>> Py_DECREF() mechanism on the PyObject* to the parent object
>>>> could be used the overhead should be pretty small. I have
>>>> tried a simple implementation of this by modifying SWIG to do
>>>> the Py_INCREF() and it appears to work well (although it is a
>>>> hack as I don't understand the SWIG internals well enough) and
>>>> I don't know how to call Py_DECREF() efficiently so I am not
>>>> doing this. I am assuming it would be possible to modify the
>>>> SWIG C++ wrapper object to have the extra container objects
>>>> pointer so its reference count can be decremented but I need to
>>>> try and understand the SWIG code better.
>>>>
>>>> If you think something like this might be feasible and useful,
>>>> I will have a look at the SWIG code a bit more in my limited
>>>> spare time.
>>>>
>>>> Terry
>>>> On 28/06/2020 13:32, William S Fulton wrote:
>>>>> Hi Terry
>>>>>
>>>>> This is one of the most horrible problem areas mixing
>>>>> languages, that is, memory management. Here is some background
>>>>> reading:
>>>>>
>>>>> http://www.swig.org/Doc4.0/Python.html#Python_memory_management_member_variables
>>>>> https://github.com/swig/swig/issues/945
>>>>> https://github.com/swig/swig/pull/1234
>>>>>
>>>>> Below is one way you can fix this (it's a variation of
>>>>> returning members by reference documented in the 1st link
>>>>> above). Note that SWIG uses pointers for accessing
>>>>> non-primitive member variables, see
>>>>> http://www.swig.org/Doc4.0/SWIG.html#SWIG_structure_data_members.
>>>>>
>>>>> %fragment("member_parent_attribute_init", "init") {
>>>>> // thread safe initialization
>>>>> member_parent_attribute();
>>>>> }
>>>>>
>>>>> %fragment("member_parent_attribute", "header",
>>>>> fragment="member_parent_attribute_init") {
>>>>> static PyObject* member_parent_attribute() {
>>>>> static PyObject* attr =
>>>>> SWIG_Python_str_FromChar("__member_parent");
>>>>> return attr;
>>>>> }
>>>>> }
>>>>> %typemap(ret, fragment="member_parent_attribute", noblock=1)
>>>>> ListA *TypeB::list1 %{
>>>>> PyObject_SetAttr($result, member_parent_attribute(), $self);
>>>>> %}
>>>>>
>>>>> William
>>>>>
>>>>> On Thu, 25 Jun 2020 at 14:58, Terry Barnaby
>>>>> <[email protected] <mailto:[email protected]>> wrote:
>>>>>
>>>>> I have had a quick look at the SWIG code.
>>>>>
>>>>> It seems that it would be possible, and desirable to do a
>>>>> Py_INCREF() on
>>>>> the PyObject* object that is wrapping the C++ object which
>>>>> you are
>>>>> returning a pointer to the member from. This would hold
>>>>> the whole 'C++'
>>>>> object in memory so that returning a pointer to one of
>>>>> this objects
>>>>> members would be ok. The PyObject* returned for the member
>>>>> would have to
>>>>> have a pointer to its parents PyObject* and do a
>>>>> Py_DECREF() on the
>>>>> parent object when it is finally deleted so that the
>>>>> original C++ object
>>>>> gets cleaned up eventually.
>>>>>
>>>>> I don't know the SWIG code to know if this is really
>>>>> possible and if
>>>>> people think this is a reasonable SWIG feature that could
>>>>> be added ?
>>>>> At the moment a user of your SWIG generated Python API
>>>>> gets a pointer to
>>>>> a destroyed object which is not good.
>>>>>
>>>>> Terry
>>>>> On 24/06/2020 10:04, Terry Barnaby wrote:
>>>>> > I have a class library that has List objects as members
>>>>> of the class.
>>>>> > If I use this class in a Python function and try to
>>>>> return one of
>>>>> > these more complex members all hell breaks loose. It
>>>>> appears the SWIG
>>>>> > generated wrapper returns a pointer to the objects
>>>>> member rather than
>>>>> > making a copy of it when returning the member from the
>>>>> function.
>>>>> >
>>>>> > So I have something like:
>>>>> >
>>>>> > class TypeB {
>>>>> >
>>>>> > public:
>>>>> >
>>>>> > List<TypeA> list1;
>>>>> >
>>>>> > };
>>>>> >
>>>>> > If in Python I have a function like the following, the
>>>>> returned item
>>>>> > is effectively (SWIG wrapped) a pointer the b objects
>>>>> member and b
>>>>> > disappears when the function returns.
>>>>> >
>>>>> > def func1():
>>>>> >
>>>>> > b = TypeB();
>>>>> >
>>>>> > b.list1.append(TypeA());
>>>>> >
>>>>> > return b.list1;
>>>>> >
>>>>> > Now I can use "return ListTypeA(b.list1)", where
>>>>> ListTypeA has been
>>>>> > declared using a %template declaration such that the
>>>>> copy constructor
>>>>> > is called to explicitly make a copy of the list, but is
>>>>> there a better
>>>>> > way of handing this ?
>>>>> >
>>>>> > I know this is difficult to handle, but perhaps
>>>>> something like a
>>>>> > %template(return) to add code when returning a type in a
>>>>> Python
>>>>> > function somehow or maybe the overall objects reference
>>>>> counter can be
>>>>> > incremented ?
>>>>> >
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > Swig-user mailing list
>>>>> > [email protected]
>>>>> <mailto:[email protected]>
>>>>> > https://lists.sourceforge.net/lists/listinfo/swig-user
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Swig-user mailing list
>>>>> [email protected]
>>>>> <mailto:[email protected]>
>>>>> https://lists.sourceforge.net/lists/listinfo/swig-user
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> Swig-user mailing list
> [email protected]
> <mailto:[email protected]>
> https://lists.sourceforge.net/lists/listinfo/swig-user
>
>
>
> --
> --------------------
> Gareth Francis
> www.gfrancisdev.co.uk <http://www.gfrancisdev.co.uk>
_______________________________________________
Swig-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/swig-user