Re: Problem with alignment under Linux [SEC=UNCLASSIFIED]

In the 32bit linux environment, which I would have expected to include a 32bit binary on a 64bit kernel, the standard packing is 4 bytes.  As far as I can tell the only basic datatype in this ABI that demands 8 byte alignment is long long - which doesn't appear directly in what you're working with...  

So there has to be something in the C headers you're working from that triggers 8 byte packing. I went back through the old thread, but didn't see any reference to any actual structure/union declarations as they appear in the C headers, so it would be useful to know how this is being triggered.

BTW, it seems to me that there's a bug in the fill_union() function as I think the union as shown should be 8 bytes rather than the 16 bytes reported - I think the calculation of the delta  should be something like

	delta  = size % 8
	if delta:
		delta = 8 - delta

to produce the correct size.

-------------------------> "These thoughts are mine alone!" <---------
Andrew MacIntyre           Operations Branch
tel:   +61 2 6219 5356     Communications Infrastructure Division
fax:   +61 2 6253 3277     Australian Communications & Media Authority
email: [email protected]            http://www.acma.gov.au/


> -----Original Message-----
> From: Axel Seibert [mailto:[email protected]]
> Sent: Thursday, 9 February 2012 12:43 AM
> To: [email protected]
> Subject: [ctypes-users] Problem with alignment under Linux
> 
> Dear list!
> 
> Please excuse the long posting.
> 
> Nearly two years ago we had a discussion here on the list about an issue that
> I have under Linux
> (http://sourceforge.net/mailarchive/forum.php?thread_name=C3A6EB6D-
> E1DD-4D91-A579-9A88F5040ED5%40globonaut.org&forum_name=ctypes-
> users).
> Unfortunately, we never resolved the issue, and I would like to take up the
> discussion once more.
> 
> Problem:
> I'm wrapping a C library from an enterprise software environment
> (ARSystem
> from BMC) with ctypes in python (I don't have access to the source code or
> any
> details about the compiler used). This is working fine under Windows and
> SPARC
> Solaris. Under Linux it's not working with the standard ctypes setup. "Does
> not work" means that I cannot use any of the library calls to talk to the
> server, as it does not receive the parameters correctly.
> 
> Assumption
> It seems that the memory alignment for this library under Linux does not
> match
> ctypes' expectations.
> 
> Workaround 1
> By using _pack_ = 8, for some of the structures and unions, I can get this to
> work reliably.
> The problem is that ctypes' implementation limits the maximum size of the
> _pack_ value. In other words, my workaround only works with a patched
> version
> of ctypes (for a patch against python 2.7.2, please see below).
> 
> Workaround 2
> During the discussion here on the mailinglist, Thomas developed two python
> workarounds, one for structures
> (http://sourceforge.net/mailarchive/message.php?msg_id=24837697), and
> one for
> unions
> (http://sourceforge.net/mailarchive/message.php?msg_id=25006328), to
> cope with my problem.
> However, with those workarounds, it still does not work.
> 
> I have now researched this a bit more, and have come up with the following
> test case:
> 
> -----------------------------------------
> #!env python
> 
> from ctypes import *
> import sys
> 
> def fill_structure(a, fields, override = False):
>     """Patch the 'fields' sequence for a ctypes Structure by inserting
>     dummy fields named '' so that the alignment of the original fields
>     is a multiple of the 'a' value.
>     """
>     if sys.platform.startswith('linux') or override:
>         result = []
>         size = 0
>         for n, t in fields:
>             size += sizeof(t)
>             missing = (a - sizeof(t)) % a
>             result.append((n, t))
>             if missing:
>                 result.append(("", c_char * missing))
>         return result
>     else:
>         return fields
> 
> def fill_union(value, fields, override = False):
>     if sys.platform.startswith('linux') or override:
>         size = 0
>         for name, typ in fields:
>             size = max(size, sizeof(typ))
>         delta = 8 - size % 8
>         return fields + [("", c_char * (size + delta))]
>     else:
>         return fields
> 
> 
> size_t = c_uint
> ARLong32 = c_long
> ARTimestamp = ARLong32
> ARTime = ARLong32
> ARULong32 = c_ulong
> 
> class N13ARValueStruct3DOLLAR_1E(Union):
>     _fields_ = fill_union(8, [
>     # ar.h 756
>     ('noval_', size_t),
>     ('keyNum', c_uint),
>     ('intVal', ARLong32),
>     ('realVal', c_double),
>     ('charVal', c_char_p),
>     ('diaryVal', c_char_p)
> ], override=True)
> 
> class ARValueStruct(Structure):
>     _fields_ = fill_structure(8, [
>     # ar.h 753
>     ('dataType', c_uint),
>     ('u', N13ARValueStruct3DOLLAR_1E),
> ], override=True)
> 
> 
> print "This is on plattform: %s" % sys.platform
> print "This is the alignment:"
> print "%14s: %7s %5s %9s" % ('Name', 'Offset', 'Size', 'Alignment')
> for name, typ in ARValueStruct._fields_:
>     if name:
>         print "%14s: %7d %5d %9d" % (name, getattr(ARValueStruct,
> name).offset, sizeof(typ), alignment(typ))
> 
> for name, typ in N13ARValueStruct3DOLLAR_1E._fields_:
>     if name:
>         print "%14s: %7d %5d %9d" % (name,
> getattr(N13ARValueStruct3DOLLAR_1E,
> name).offset, sizeof(typ), alignment(typ))
> 
> -----------------------------
> 
> If you run this under Linux, it will print out the following result:
> This is on plattform: linux3
> This is the alignment:
>           Name:  Offset  Size Alignment
>       dataType:       0     4         4
>              u:       8    16         4
>         noval_:       0     4         4
>         keyNum:       0     4         4
>         intVal:       0     4         4
>        realVal:       0     8         4
>        charVal:       0     4         4
>       diaryVal:       0     4         4
> 
> 
> When I take the Structure and Union definition, add a _pack_ = 8 and use my
> patched version of ctypes, I receive the following (please note the different
> output for the field 'u'):
> This is on plattform: linux3
> This is the alignment:
>           Name:  Offset  Size Alignment
>       dataType:       0     4         4
>              u:       8    16         8
>         noval_:       0     4         4
>         keyNum:       0     4         4
>         intVal:       0     4         4
>        realVal:       0     8         4
>        charVal:       0     4         4
>       diaryVal:       0     4         4
> 
> I'm asking the experts on this list to try to understand my problem. If you
> are missing any details or additional information, please do not hesitate to
> contact me. I'm also willing to run any kinds of tests and stuff. I would very
> much appreciate any python based solution. The current situation, that users
> of the library need to patch Python cannot be the solution.
> 
> Thanx for your time & help!
> Axel
> 
> 
> Patch for python 2.7.2
> *** Modules/_ctypes/cfield.c.orig	2012-02-05 12:56:21.902652940 +0100
> --- Modules/_ctypes/cfield.c	2012-02-05 12:56:41.770652136 +0100
> ***************
> *** 150,156 ****
>           /* fall through */
>       case NO_BITFIELD:
>           if (pack)
> !             align = min(pack, dict->align);
>           else
>               align = dict->align;
>           if (align && *poffset % align) {
> --- 150,156 ----
>           /* fall through */
>       case NO_BITFIELD:
>           if (pack)
> !             align = pack; /*min(pack, dict->align);*/
>           else
>               align = dict->align;
>           if (align && *poffset % align) {
> 
> 
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> ctypes-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/ctypes-users

NOTICE: This email message is for the sole use of the intended recipient(s) 
 and may contain confidential and privileged information. Any unauthorized 
 review, use, disclosure or distribution is prohibited. If you are not the 
 intended recipient, please contact the sender by reply email and destroy all 
 copies of the original message.

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/