Re: Loadable module support

On Fri, 2012-01-20 at 17:20 +0200, Boris Kolpackov wrote:
> >         load <name> [<pathname>]
> > 
> > The semantics would be something like this:
> > 
> >      1. See if the symbol "gmk_<name>_init" (or something?) is defined.
> >      2. If not, then loop through the following paths in order:
> >         <pathname> (if given), or the expansion of <name> against a list
> >         of patterns in a variable such as .LOADPATTERNS, the default
> >         value of which would be something like "%.o lib%.so".
> >      3. For each one in turn, try loading it with dlopen().  Should we
> >         support VPATH search algorithms for these as well?  Do we want
> >         to try to suppress normal dlopen() search algorithms
> >         (LD_LIBRARY_PATH etc.)?
> 
> This sounds overly complex to me. dlopen() (not sure about Windows
> LoadLibrary()) doesn't really care about what the name of the module
> is. For example, in the GCC plugin architecture you pass a full path
> to the name of the module to load. GCC doesn't search LD_LIBRARY_PATH
> or append any prefixed (like lib) or suffixes (like .so) to it. Maybe
> there is some value in all this (like writing portable modules) though
> I am not sure how useful this will be since building a shared object/DLL
> is already not portable. Can you maybe give a motivating example for
> these features?

dlopen() will not search if the pathname you give it contains slashes.
If it doesn't, dlopen() will look in a number of "standard" locations;
the details are in the man page.

What I wanted to do was allow something simple like:

	load foobar

And have it all just work.  I need two things for loading: the name of
the dynamic object file and the name of the symbol for "init".  I
thought it would be nice to allow a single token on the "load" line and
be able to infer both those things from that one token.  In order to do
that I need to be able to translate the token into both a symbol (easy)
and the object to be loaded... hence step #2 above.

Also this allows some portability (foobar.so vs. foobar.dll or whatever
can be abstracted).

But maybe that's too nice and we should just force people to provide
both items: the object name and the symbol (or token).

The idea behind VPATH is that, just like make include files, people
might have pre-built modules in some central location in their tree and
they'd like to be able to define that location in a simple way (users
are always asking why include doesn't search VPATH).  But again, maybe
that's too much.

> > My intention is to not make any extra effort (beyond defining an
> > interface for registering new functions) to provide any defined internal
> > interfaces for these modules.  If they want to take advantage of some
> > other functions in make code they can but there is no guarantee that
> > they will exist, unchanged, in the next release of GNU make.
> 
> Yes, I think this is a good approach. However, to allow the plugin
> to do more we will need to install the make headers. I would suggest
> the /usr/include/make/ directory for that

Hm.  Good point.  Ouch.  Those headers are in no shape to be
published :-).  There will have to be huge disclaimers about the fact
that we are not guaranteeing any of these interfaces are stable, in any
way.  Over time maybe we can create some public, stable APIs but I'm not
ready to commit to any of those now... that means that custom modules
could break with every new version of GNU make.

Well, I guess I can commit to having the "define a new function" API be
stable :-).

> (I would also use the 'make' prefix instead of 'gmk' in function
> names; or 'gmake' if 'make' feels too generic).

Hm.

> Also, it would be good to define a make variable which contains the
> path to the directory containing the headers, e.g., .MAKE_HEADERS.
> So that we can add the necessary -I option when building the module.

Well, the best we can do here is give the path to the place that we
thought those headers would go when make was configured.  We can't, for
example, take into account changes at install time by "make install
prefix=..." or "make DESTDIR=", much less any post-processing (chroots,
cross-compilation environments, etc.) etc.

I'm not sure this is helpful to be honest.  The standard install will
put the headers in a directory that will be automatically searched by
the compiler under normal circumstances (e.g., /usr/include) so in that
case the -I is not needed, and if they're not there then most likely the
person writing the makefiles knows where they'll be anyway.

What do you think?