Foreign function interface

I don't have a concrete proposal at this time.  I'm just musing idly
about something that would be nice to have.  I'm also going to wish for
a magic wand down below.  If you happen to be the owner of such a wand,
please speak up!

The loadable module feature of XEmacs, which I maintain, has some
drawbacks:

  - Modules cannot be included in packages.  It might be possible to
    remedy this for Unix platforms, but will always be a pain for
    platforms where we cannot assume that the target machine has an
    installed compiler.

  - Two shared objects must be loaded for every one the user wants: the
    actual library, and an ellcc-produced object file that provides the
    glue between XEmacs and the shared library.  This eats up file
    descriptors at double the optimal rate.

  - We have to have a special compiler wrapper, ellcc, to produce the
    glue objects.  That wrapper requires a real compiler to wrap, and
    has never been ported to native Windows in spite of multiple
    attempts to find someone willing to do the work over a span of a
    couple of years.

  - Our compiler wrapper was written when libtool was young.  In effect,
    it recapitulates all the work that has gone into libtool, but has
    missed out on libtool advancements / enhancements over the last
    several years.

  - Making modules work on Cygwin (and eventually native Windows)
    requires dealing with the Windows linker.  That is why we now have
    MODULE_API all over the place.  However, it is hard to predict what
    future modules will need to access.

Wouldn't it be nice if we had a flexible foreign function interface
instead?  That would eliminiate all of these problems.  The code that is
currently written as modules would be rewritten as straight Lisp code
(and therefore be packagizable [1]) that uses some special functions to
find the relevant shared object and put Lisp wrappers around the
functions and variables it contains.  You load only the shared objects
that are absolutely required, no compiler is necessary, ellcc is not
needed, etc.  Bill Perry did something like this with the GTK code, but
didn't go far enough for general use.  (In fact, he couldn't even handle
all of the GTK functions automatically; some had to be wrapped by hand.)

Wrapping variables shouldn't be terribly difficult, but macros will be
pretty much impossible to deal with.  Also, I need a magic wand to turn
a function pointer (such as returned by dlsym) and some parameters into
a C function call.  Unfortunately, this is messy.  Some register-poor
platforms, like the x86, pass all function parameters on the stack
(except for the ccall/stdcall distinction, of course).  Some pass some
parameters in registers and the rest on the stack.  The rules for which
parameter to pass where can get pretty complicated.  The rules for
MIPS/Irix, for example, are quite hairy, and gcc didn't get them all the
way right for many years; see the discussion on passing structs as
parameters and return values in the GCC docs.

Various people have tried to encapsulate the information needed to
construct C calls on the fly.  For example, the GCC source tree contains
a libffi subdirectory.  However, we would need to investigate its
copyright and license status before using it [2,3].  It supports a fair
number of CPU/OS combinations (various OSes using alpha, arm, ia64,
m68k, mips, powerpc, s390, sh, sh64, sparc, and x86 at the moment).  It
has been used to build foreign function interfaces for other languages
and packages, such as Python[4], Tcl[5], Java (e.g., Kaffe[6], gcj[7],
and SableVM [8]), Guile[9], and GNUstep[10].

There is also Bruno Haible's GPLed ffcall package [11] which supports
the same CPUs as libffi EXCEPT for s390, sh, and sh64, and additionally
supports convex, hppa, m88k, and x86_64.  Bruno uses it to provide the
foreign function interface in CLISP.  It is also used by GNUstep[10], and
provides the foreign function interface for Ruby[12], Forth[13], and
PLT-Scheme[14].

On the other hand, all that really matters is whether a given
parameter/return type is a 1-, 2-, 4-, 8-, ... byte integer, or a float,
double, or long double, a pointer to something, an ellipsis, or a
union/struct.  We cannot deal with unions and structs in a general way,
so ignore them.  It would not be too difficult to automatically generate
every combination of the others, from 0 parameters out to some
combinatorially manageable maximum.  Then you just cast your dlopen()
returned pointer to the type the wrapper wants and away you go with no
external library needed.  If you want to use a function that doesn't
match one of the automatically generated prototypes, you are out of
luck.  You cannot use the foreign function interface; you must use the
module interface.

Anyhow, it might be useful to see how other folks constructed their FFIs
and see if we can scheme up something [15] that would be good for
XEmacs.  I think this would be much easier to use than the module
interface, and would make it more likely that wrappers for C libraries
will be written.

Thoughts?

Footnotes:
 [1] I nominate "packagizable" for the Neologism of the Week Award.
 [2] http://gcc.gnu.org/ml/gcc-patches/2003-08/msg01784.html
 [3] http://sources.redhat.com/ml/libffi-discuss/2004/msg00000.html
 [4] http://starship.python.net/crew/theller/ctypes/
 [5] http://www.elf.org/ffidl/
 [6] http://www.kaffe.org/pipermail/kaffe/2004-August/099512.html
 [7] http://gnu.blic.net/software/gcc/java/
 [8] http://sablevm.org/
 [9] http://www.gnu.org/software/guile/guile.html
[10] http://www.gnustep.org/
[11] http://www.haible.de/bruno/packages-ffcall.html
[12] http://web.sfc.keio.ac.jp/~s01397ms/RWiki/index.cgi?cmd=view;name=ruby-ffcall
[13] http://www.jwdt.com/~paysan/gforth.html
[14] http://www.cs.brown.edu/research/plt/
[15] Pun unashamedly intended.
-- 
Jerry James
http://www.ittc.ku.edu/~james/