Re: Registries

At 20:23 2003-07-17, Ralf W. Grosse-Kunstleve wrote:
> >> So is it a *real* problem? I am not sure. It could become a little bit
> >> of a nightmare if the number of packages using Boost.Python keeps
> >> growing. People will start using them in unforeseen combinations
> >> and there will likely be conflicting bindings for types such as
> >> std::vector<> or std::map<>.
> >
> > I think the answer to that is to make it much easier and more obvious
> > to link statically to the langbinding library or to use a seprate
> > copy of the .so when building packages designed for CBD.
>
>Consider:
>
>- Development team A releases package_a
>- Development team B releases package_b
>
>The development teams do not know about each other.
>
>- User U has an application using binary distributions of
>   both package_a and package_b.
>
>package_a has bindings for std::vector<double> <-> Python tuple.
>package_b uses class_<std::vector<double> >.
>
>What would the teams A and B have to do to make this work on all
>platforms? Remember that Tru64 puts all symbols in one "link
>namespace" (not sure what the correct term is here).

Is there any problems with shipping their extensions with separate .so's or
linking statically?

>Now User U writes his own extension module using Boost.Python. He
>wants to use the std::vector<> bindings provided by package_b. What
>does he have to do?

Imports the converters from package_b explicitly, or links to package_b's
From nobody Sat Jul 19 11:06:16 2003
Return-Path: <[email protected]>
Received: by stlport.com (CommuniGate Pro PIPE 3.5.9)
  with PIPE id 285030; Thu, 17 Jul 2003 12:01:11 -0700
Received: from redshift-software.com ([12.100.89.42] verified)
	by stlport.com (CommuniGate Pro SMTP 3.5.9)
	with ESMTP id 285029 for [email protected];
	Thu, 17 Jul 2003 12:01:10 -0700
X-AuthUser: [email protected]
Received: from 12.100.89.43 (12.100.89.43)Server]
	<[email protected]>;	Thu, 17 Jul 2003 14:01:10 -0500
Date: Thu, 17 Jul 2003 14:01:21 -0500
From: Rene Rivera <[email protected]>
Subject: Re: [Boost-langbinding] Registries
To: David Abrahams <[email protected]>, 
    [email protected]
X-Priority: 3
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; Charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Mailer: Mailsmith 1.1.8 (Bluto)
X-Spambayes-Classification: ham; 0.00
Lines: 196
Xref: PENGUIN langbinding-mbox:78
X-Gnus-Newsgroup: langbinding-mbox:78   Sat Jul 19 11:06:16 2003

[2003-07-17] David Abrahams wrote:

>Rene Rivera <[email protected]> writes:
>
>> [2003-07-17] David Abrahams wrote:
>>
>>>I think that most OSes have a mechanism where you can specify exactly
>>>which symbols are shared, but for C++ at least, on OSes other than
>>>windows, it's usually much too cumbersome to be worthwhile and
>>>requires knowledge of name mangling.  If I'm wrong about that I hope
>>>someone will correct me (Rene?).
>>
>> Not totally wrong ;-) Doing the equivalent of DLLEXPORT on Linux is
>> something I have built into the build process of my project. Does not
>> require knowledge of the name mangling. What it does require is platform
>> specific linker knowledge, and access to tools that can manipulate
linkable
>> objects. Those requirements are usually around for ELF type system (not
just
>> Linux). All that said, it's probably still too cumbersome for general
use.
>>
>> Looking at the handy shared lib guide...
>>
>> http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html
>> Using static and shared libraries across platforms
>>
>> Gives you some idea of other methods. The ones that mention doing a
>> script file for "ld" are the hard ones to implement.
>
>How hard is hard?  Can we see your script?

Below is the part in my jamrules (BBv1) file that does the setup for it. But
first some explanation. It relies on the programmer marking the symbols that
one wants exported, in a similar way to what win32 does with
__declspec(dllexport). It uses this GCC option on the symbols that want to
get exported: __attribute__ ((section (".text.dll"))) Although that's a GCC
only feature the only real need of the script is to have a way of building
what the exported set of symbols are. I use a different section because it's
easy to automaticaly put in the section attribute in the code (at least as
easy as dllexport). I'm adding some additional comments below in [[]]...

[[ I use a <globals-segment>".text.dll" on the BB targets that I want to do
the export symbol filtering. For me I only do this on release targets, as I
want to be able to have all the possible debug symbols otherwise. ]]

# Sepcify the segment to filter by for "obj" targets.
#
free-feature globals-segment ;
for local _T_ in $(TOOLS)
{
        flags $(_T_) GLOBALS_SEGMENT <globals-segment> ;
}

[[ The way BBv1 works the easy way to do this is have a new target type that
does the additional steps of filtering on the link stage. ]]

# Declare a shared object target.
#
SHARED_TYPES += SO ;
gTARGET_TYPE_ID(so) = SO ;
gGENERATOR_FUNCTION(SO) = shared-obj-file ;
gIS_DEPENDENCY(SO) = TRUE ;
SOMODE = $(DLLMODE) ;
SUFSO = $(SUFDLL) ;
PRESO = $(PREDLL) ;
gTARGET_TYPE_REQUIREMENTS(SO) = <shared-linkable>true ;
rule so ( target : sources + : requirements * : default-build * )
{
        declare-local-target $(target) : $(sources) : $(requirements) :
$(default-build) : SO ;
}

# obj: generator function
#
rule shared-obj-file
{
        type-DEPENDS so : $(<) ;
        main-from-objects $(<) : [ Objects $(>) ] : SO ;
}

[[Here I check the globals-segment feature to see if I do the complex filter
or just the simple one step link. The complex link side does the linking in
two stages: 1) pre-link without regular startup code but with filtering of
symbols, 2) re-link with the previous result and the additional gcc/ld/C+++
requirements. Only #1 is the part where the filtering happens, see further
below. ]]
#
rule Link-SO ( target * : sources * : type )
{
        flags $(gCURRENT_TOOLSET) GLOBALS_SEGMENT <globals-segment> ;
        if $(GLOBALS_SEGMENT) && $(UNIX)
        {
                local lo-target = $(target:S=.lo) ;
                MakeLocate $(lo-target) : $(LOCATE_TARGET) ;
                Clean clean : $(lo-target) ;
                MODE on $(lo-target) = $(FILEMODE) ;
                
                NEEDLIBS on $(lo-target) = $(NEEDLIBS) ;
                DEPENDS $(lo-target) : $(sources) $(NEEDLIBS) ;
                GLOBALS_SEGMENT on $(lo-target) = $(GLOBALS_SEGMENT) ;
                Link-SO-filter-action $(lo-target) : $(sources) : $(type) ;
                
                Chmod $(lo-target) ;
                
                NEEDLIBS on $(target) = ;
                DEPENDS $(target) : $(lo-target) ;
                Link-DLL $(target) : $(lo-target) ;
        }
        else
        {
                Link-DLL $(target) : $(sources) ;
        }
}

#
actions Link-SO-action bind NEEDLIBS
{
    ld -nostdlib -nostartfiles -Ur "$(>)" $(NEEDLIBS) -o "$(<)"
}

[[ Entertaining aspect of Linux is that the regular wgrep doesn work at all
as advertised. So I had to improvise with some custom multi calls to the
regular grep. This also make the script below shorter ;-) ]]

WGREP = "while read SYM ; do grep -w -F \"--regexp=${SYM}\" \"${F}\" ; done"
;
TGREP = "while read SYM ; do grep \"--regexp=${SYM}$\" \"${F}\" ; done" ;

[[ OK the real filtered link script (sorry for the email line wrap) The
basic algo for this is: 1) do a bare link step, 2) extract the export
symbols into an export file, 3) copy the temporary link object while
filtering to only make public the symbols in the export file. ]]
#
actions Link-SO-filter-action bind NEEDLIBS
{
[[ #1 Does a link of the source objects into one searchable re-linkable
object. The relinkable part is the "-Ur" option. ]]
    ld -nostdlib -nostartfiles -Ur "$(>)" $(NEEDLIBS) $(NEEDLIBS) -o "$(<)"
[[ #2-a Does a dump of all the symbols. ]]
    objdump -t -w "$(<)" > "$(<:S=.dump)"
[[ #2-b Does a series of dumps with C++ name demangling to be able to cross
reference from the unmangle export symbols to the vtables and typeinfo
generated, to the mangled symbol names.]]
    objdump -C -t -w "$(<)" | grep -F 'vtable' > "$(<:S=.vtable)"
    objdump -C -t -w "$(<)" | grep -F 'typeinfo for' > "$(<:S=.typeinfo)"
[[ #2-c This dumps all the unmangled symbols and filters based on wether if
find the segment for that symbol to be the export only one. The result here
is one file with all the symbols that are intended for public use
(unmangled).]]
    objdump -C -t -w "$(<)" | grep -F '$(GLOBALS_SEGMENT)' | sed
's/^....................................//g' | sed 's/[(].*$//g' | sed
's/::[^:]*$//g' | sort | uniq > "$(<:S=.classes)"
[[ #2-d We also need to cross-reference that ouput with the output of vtable
and typeinfo symbols to also export the vtables+typeinfo for any classes
that are referenced. ]]
    cat "$(<:S=.classes)" | sed 's/^.*$/vtable for &/g' | (
F="$(<:S=.vtable)" ; $(TGREP) ) | sed 's/^[^ ]*[ ]*[^ ]*[ ]*[^ ]*[ ]*//g' |
sed 's/..........vtable.*//g' > "$(<:S=.vexp)"
    cat "$(<:S=.classes)" | sed 's/^.*$/typeinfo for &/g' | (
F="$(<:S=.typeinfo)" ; $(TGREP) ) | sed 's/^[^ ]*[ ]*[^ ]*[ ]*[^ ]*[ ]*//g'
| sed 's/..........typeinfo.*//g' > "$(<:S=.texp)"
[[ #2-e And now that we have the lists of symbols we are exporting we need
to cross-reference back into the mangled names. ]]
    cat "$(<:S=.dump)" | grep -F '$(GLOBALS_SEGMENT)' | sed -e
's/^....................................//g' > "$(<:S=.exp)"
    cat "$(<:S=.vexp)" | ( F="$(<:S=.dump)" ; $(WGREP) ) | sed
's/^.................[^ ]* //g' >> "$(<:S=.exp)"
    cat "$(<:S=.texp)" | ( F="$(<:S=.dump)" ; $(WGREP) ) | sed
's/^.................[^ ]* //g' >> "$(<:S=.exp)"
[[ #3 Finally we copy (over writting the original link object from #1) only
keeping the exported symbols as public. All other symbols are stripped out
or marked private (in the ELF symbol table). ]]
    objcopy --strip-unneeded --keep-global-symbols="$(<:S=.exp)" "$(<)"
    $(RM) "$(<:S=.dump)" "$(<:S=.exp)" "$(<:S=.vtable)" "$(<:S=.classes)"
"$(<:S=.vexp)" "$(<:S=.typeinfo)" "$(<:S=.texp)"
}

[[ NOTE: Above is Copyright (C) 2001-2003 Redshift Software, Inc. --
Although feel free to use the concepts I did, but not the code ;-) ]]

All that said, the requirements for the above to work aren't all that great
but it is a complex set of operations. The tools required are those found in
a reasonably up to date version of RH73. Specifically if there's a
reasonably up to date version of GNU-binutils you'll have the objdump and
objcopy tools with the needed functionality in them (the --strip-unneeded
and --keep-global-symbols).

So you tell me... Is it worth considering something like or not? But.. I
certainly won't push for such a solution.


-- grafik - Don't Assume Anything
-- rrivera (at) acm.org - grafik (at) redshift-software.com
-- 102708583 (at) icq