Re: LP_c_ubyte buffer and length to python string?

Dan Stromberg wrote, On 07/24/2011 09:09 PM:
> Is it the case that ctypes variables are garbage collected, as long as 
> the functions you're calling don't do their own malloc'ing or other 
> form of memory allocation?

ctypes objects are Python objects and have a lifetime like other Python 
objects (ie reference counting and occasional garbage collection in 
c-python), so you have to make sure you keep a reference to the objects 
as long as you reference their memory.

>
>     You might want to post a minimal self-contained code snippet that
>     shows what function you are calling and how you are doing it.
>
>
> I'm calling 3 functions:
> 1) One gets an upper bound on how big the compressed size of a block 
> can be
> 2) The next compresses a block of data
> 3) The last uncompresses a block of data
>
> I'm not sure how self-contained this is, but here's the class.

This class wrapping is definitely not minimal ...

>     @classmethod
>     def declare_c_function(cls, library, name, argtypes=None, 
> restype=None):
>         '''Extract functions from liblzma'''
>         try:
>             func = getattr(cls.LZMA_LIB, '_%s' % name)
>             cls.funcs[name] = func
>         except AttributeError:
>             func = getattr(cls.LZMA_LIB, name)
>             cls.funcs[name] = func
>         if argtypes is not None:
>             func.argtypes = argtypes
>         if restype is not None:
>             func.restype = restype
>
>     @classmethod
>     def class_init(cls):
>         cls.LZMA_LIBPATH = _find_lib('lzma')
>         cls.C_LIBPATH = _find_lib('c')
>
>         # *ix way - windows is different
>         cls.LZMA_LIB = ctypes.CDLL(cls.LZMA_LIBPATH)
>         cls.C_LIB = ctypes.CDLL(cls.LZMA_LIBPATH)
>
>         cls.declare_c_function(cls.LZMA_LIB, 
> 'lzma_stream_buffer_bound', (ctypes.c_size_t, ), ctypes.c_size_t)

Why not just use the more simple pattern from 
http://docs.python.org/library/ctypes#return-types ? Not that I care, 
but I would recommend keeping it simple. ;-)

>         # lzma_easy_buffer_encode notes:
>         # Argument 1, uint32_t preset:
>         #    Just a uint32 - simple
>         # Argument 2, lzma_check check:
>         #    An enum to the C programmer - that should usually be an 
> unsigned integer in the C runtime.
>         # Argument 3, lzma_allocator *lzma_allocator:
>         #    This is really a pointer to a struct, but happily, we 
> only need to pass NULL to it, so we can just treat it as a void *
>         # Argument 4: uint8_t *in:
>         # Argument 5: size_t in_size:
>         # Argument 6: uint8_t *out
>         # Argument 7: size_t *out_pos
>         # Argument 8: size_t out_size
>         #
>         # The return type is also an enum, so probably an unsigned int
>         c_size_t_p = ctypes.POINTER(ctypes.c_size_t)
>         cls.declare_c_function(
>             cls.LZMA_LIB,
>             'lzma_easy_buffer_encode',
>                 (
>                 ctypes.c_uint32,                    # preset
>                 ctypes.c_uint,                      # check
>                 ctypes.c_void_p,                    # lzma_allocator
>                 ctypes.POINTER(ctypes.c_uint8),     # uint8_t *in
>                 ctypes.c_size_t,                    # size_t in_size
>                 ctypes.POINTER(ctypes.c_uint8),     # uint8_t *out
>                 ctypes.c_void_p,                    # size_t *out_pos 
> # ctypes made size_t * think it was a long, so we use void_p
>                 ctypes.c_size_t,                    # size_t out_size
>                 ),
>             ctypes.c_uint,
>             )

>     @classmethod
>     def compress(cls, input_data):
>         '''Compress data into xz format using ctypes to access 
> liblzma.so'''
>         # maximum_size = lzma_stream_buffer_bound(input_buffer_size);
>         length_input_data = len(input_data)
>         maximum_size = 
> cls.funcs['lzma_stream_buffer_bound'](length_input_data)
>
>         # This is an efficient way of creating a readonly ctypes string
>         ctypes_input_data_char_p = ctypes.c_char_p(input_data)
>         ctypes_input_data = ctypes.cast(ctypes_input_data_char_p, 
> ctypes.POINTER(ctypes.c_ubyte))

Note that most of this casting and conversion is done automatically when 
argtypes is specified correctly.

Specify this function parameter type as c_char_p and then you can give 
it input_data directly.

>         # This is a less-fast but mutable way of creating a ctypes string
>         ctypes_compressed_buffer_char_p = 
> ctypes.create_string_buffer(maximum_size)
>         ctypes_compressed_buffer = 
> ctypes.cast(ctypes_compressed_buffer_char_p, 
> ctypes.POINTER(ctypes.c_ubyte))

Specify this function parameter type as c_char_p and then you can give 
it the string buffer directly.

>         ctypes_compressed_size = ctypes.c_size_t(0)
>         #ctypes_compressed_size_pointer = ctypes.POINTER(ctypes.c_size_t)
>         ctypes_compressed_size_pointer = 
> ctypes.cast(ctypes.addressof(ctypes_compressed_size), ctypes.c_void_p)

Specify this function parameter type as POINTER(c_size_t) and then you 
can give it byref(ctypes_compressed_size).

>
>         # lzma_easy_buffer_encode
>         ret_xz = cls.funcs['lzma_easy_buffer_encode'](
>           cls.LZMA_PRESET_DEFAULT,
>             cls.LZMA_CHECK_CRC32,
>             None,
>             ctypes_input_data,
>             length_input_data,
>             ctypes_compressed_buffer,
>             ctypes_compressed_size_pointer,
>             maximum_size,
>             )
>         if ret_xz != cls.LZMA_OK:
>             raise OSError(cls.get_xz_error(ret_xz))
>
>         resultant_length = int(ctypes_compressed_size.value)

You have already specified the return type as int so there is no reason 
to specify it again here.

>         result = ctypes_compressed_buffer_char_p.raw[:resultant_length]
>
>         return result

/Mads

------------------------------------------------------------------------------
Magic Quadrant for Content-Aware Data Loss Prevention
Research study explores the data loss prevention market. Includes in-depth
analysis on the changes within the DLP market, and the criteria used to
evaluate the strengths and weaknesses of these DLP solutions.
http://www.accelacomm.com/jaw/sfnl/114/51385063/