Re: Array allocation

"Diez B. Roggisch" <[email protected]> Thu, 25 Sep 2014 12:53:39 +0200
Newsgroups	gmane.comp.python.ctypes
Message-ID	<[email protected]>
Hi,

this is more a design than a technical question. The status quo is pretty clear: the DLL allocates, and then nobody deallocates - so you are leaking.

You can of course use Python to deallocate the data, by simply calling the free-function from the libc, or the equivalent on windows - which I do not know… If push comes to shove, implement myfree in your library with the same signature, and just make that call the OS' free.

But to me that is kinda crude. The best thing to would be to take advantage of Python's GC, because then you can completely forget about managing the memory yourself.

If your given example describes the nature of the problem sufficiently (meaning you *always* allocate a constant amount of data), you could make it a requirement to pass that amount of data in.

An alternative could be to return a struct, with the 20-byte-array as a member. This should have copy-semantics in C, so the returned Python-object+associated "raw" data will be deallocated by the GC.

When you are dynamic in your size-requirements, you can use API-designs as used by the OS, e.g. fread and similar: you pass not only in a pointer to a buffer, but *also* it's length, and return the amount of actually used bytes.

With this, you make it explicit in your interface design what the programmer has to do. Of course this allows for error-cases where you provide less data than required, and need to check for them. But that could be done once, in a Pythonic wrapping you provide.

HTH,

Diez


On Sep 24, 2014, at 6:15 AM, Koteswara Rao Ruvva <[email protected]> wrote:

> There was a mistake in my DLL and corrected. It works great. No changes were made to the code written by Eryksun. Wonderful.
> 
> BTW, how do we deallocate the memory (data) after its use. Is there a way to free that memory in Python code? Or should that be done in DLL? 
> 
> I would like to make this problem complete in the thread so that Python users can have clear picture from end to end.
> 
> From: Koteswara Rao Ruvva <[email protected]>
> To: eryksun <[email protected]> 
> Cc: "[email protected]" <[email protected]> 
> Sent: Monday, September 22, 2014 10:27 PM
> Subject: Re: [ctypes-users] Array allocation
> 
> Great. Very well explained. 
> 
> For some reason, it is crashing with no erros in the console soon after mydll.getData(ctypes.byref(data)) is called. Reviewed the C code in the DLL carefully and found no issues.
> 
> I am on Windows 7 64Bit system with Python 2.7.7
> 
> Do you have any clue?
> 
> Thanks
> Kotesh
> 
> 
> From: eryksun <[email protected]>
> To: Koteswara Rao Ruvva <[email protected]> 
> Cc: "[email protected]" <[email protected]> 
> Sent: Monday, September 22, 2014 5:51 PM
> Subject: Re: [ctypes-users] Array allocation
> 
> On Mon, Sep 22, 2014 at 12:34 AM, Koteswara Rao Ruvva <[email protected]> wrote:
> > I have the following getData function in a DLL that allocates memory and
> > fill its contents.
> >
> > extern "C" __declspec(dllexport) int getData(int *data) {
> >    int size = 20;
> >    data = (int*) malloc(size*sizeof(int));
> >
> >    for (int i = 0; i < size; i++) {
> >        data[i] = i*i;
> >    }
> >
> >    return size;
> > }
> >
> > I have the following Python code to call the getData function:
> >
> >        mydll = ctypes.cdll.LoadLibrary('MathFuncs.dll')
> >
> >        mydll.getData.argtypes = [ctypes.POINTER(ctypes.c_int)]
> >        mydll.getData.restype  = ctypes.c_int
> >
> >        data = ctypes.POINTER(ctypes.c_int)
> >        size = mydll.getData(ctypes.POINTER(data))
> >
> > When I execute, I get the following error:
> > ctypes.ArgumentError: argument 1: <type 'exceptions.TypeError'>: expected
> > LP_c_long instance instead of _ctypes.PyCPointerType
> >
> > I am not able to figure out the mistake. Could someone please correct me.
> 
> POINTER returns a pointer type that can be set in a function's
> argtypes. Subsequently calling the function requires that the
> corresponding argument is an instance of the type. So the immediate
> problem here is that `data` needs to be an instance of the pointer
> type.
> 
> That said, don't pass the pointer by value (initially NULL). Instead
> pass by reference in order to receive the address of the malloc'd
> buffer. For example:
> 
> // mathfuncs.cpp
> 
> #include <stdlib.h>
> 
> // The data paramater has to be a pointer to a pointer, i.e.
> // an int **. The address of the malloc'd buffer gets stored
> // to the caller's pointer, i.e. *data.
> 
> extern "C" __declspec(dllexport) int getData(int **data) {
>     int size = 20;
>     int *arr = (int *)malloc(size * sizeof(int));
>     for (int i = 0; i < size; i++)
>         arr[i] = i*i;
>     *data = arr;
>     return size;
> }
> 
> 
> # test_mathfuncs.py
> 
> import ctypes
> 
> # Use CDLL instead of cdll.LoadLibrary. Windows appends the .dll
> # extension for you.
> mydll = ctypes.CDLL('mathfuncs')
> 
> c_int_p = ctypes.POINTER(ctypes.c_int)
> c_int_pp = ctypes.POINTER(c_int_p)
> 
> mydll.getData.argtypes = [c_int_pp]
> mydll.getData.restype  = ctypes.c_int
> 
> # Instantiate an int * pointer.
> data = c_int_p()
> 
> # Use byref to pass the address of the pointer.
> size = mydll.getData(ctypes.byref(data))
> 
> print("data.contents is", data.contents)
> # data.contents is c_long(0)
> 
> print("size ==", size)
> 
> # size == 20
> 
> 
> print("data[size-1] ==", data[size-1])
> # data[size-1] == 361
> 
> print("data[:3] ==", data[:3])
> # data[:3] == [0, 1, 4]
> 
> # Casting the result to an array requires
> # a pointer cast to int (*)[size]; then
> # a pointer dereference.
> # typedefs
> array_t = ctypes.c_int * size
> array_t_p = ctypes.POINTER(array_t)
> # pointer cast
> p_data_array = ctypes.cast(data, array_t_p)
> # pointer dereference
> data_array = p_data_array.contents
> 
> # array_t is an aggregate type consisting of 20 elements
> # that are each 4 bytes.
> 
> print("sizeof(data_array) ==",  ctypes.sizeof(data_array))
> print("len(data_array) ==", len(data_array))
> # sizeof(data_array) == 80
> # len(data_array) == 20
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------------
> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
> ctypes-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/ctypes-users

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk

_______________________________________________
ctypes-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ctypes-users
signature.asc (application/pgp-signature, 496 B)