WG: The (floating) address of foreign arrays (FFI talk)

Hi,
[the same topic arose independently on both clisp and uffi mailing lists. I thus write to both.]

When a C programmers calls
 x = foo(...,pointer or &array, ...) s/he defines an implicit contract:
 1) the pointer points to data that will be valid at the time it will be needed.
E.g. it may be a reference to an array to be filled up by the call, or installing an i/o buffer or callback address for later use.

When a Java/Lisp/Smalltalk programmers invokes a foreign function, the contract is slightly different:
 2) the pointer into a Lisp/Java object is valid for the duration of the call.

Even the weaker contract 2) can be endangered in implementations that provide
 i) a moving (compacting) GC *and*
 ii) callbacks into Lisp/Java or programmable signal handlers

E.g. suppose a signal handler or a callback is invoked while in foo. The handler calls Lisp code, which causes a GC. The address of the double float array passed to foo may not be valid anymore, leading to random crashes.


One might be tempted to ask: "what the heck? in my code I don't have a handler!".
My answer is that only the system architect, s/he who knows
the complete system can hold such a claim about the system as a whole.
The individual component writer cannot know whether some
other component installed e.g. some signal handler in Lisp.


Ignoring this issue yields to unreliable "it works for me" solutions.
I'm opposed to simple looking solutions that only work in limited,
unspecified contexts. Such solutions are obviously too simple than
possible (w.r.t. Albert Einstein's famous quote).


There are several solutions:
 a) mark Lisp objects passed as reference to be non-movable (during the foreign call), or
 b) don't pass pointers to Lisp objects, use space on the
    execution stack instead. That's what CLISP (foreign-call-out,
    with-foreign-object), CMUCL (with-alien) and
    other implementations do, or
 c) use a non-moving GC, e.g. Boehm's conservative GC. However doing
    so precludes nice GC design, probably generational GC and other
    modern features of advanced GC schemes, or
 d) add explicit declarations to the effect that the given foreign
    call (e.g. memcpy) is free of callbacks (while XmainLoop() is not).
 e) Give guarantees and a model about the context in which Lisp-level
    signal handlers are called by the implementation (it's typically
    delayed outside the signal handler). E.g. in CLISP, they could be
    invoked by the bytecode interpreter loop.
 f) forbid passing addresses of Lisp objects to functions not declared
    free of callbacks in implementations where Lisp-level signal handlers
    can be called anytime.
 g) ...

I hope this made very clear why I claim that the limited declarative power of the popular Allegro construct
(ff:def-foreign-call half-double-vector
    ((size :int)
     (vec (:array :double)))
  :returning :void)
where vec is used for i/o and writes directly to a Lisp double float array is dangerous -- outside of Allegro. Such a declaration cannot be the base for a UniversalFFI. One has to consider i) and ii) above.

That's also why there must *not* be a (WITH-LISP-ARRAY-ADDRESS (addr-var array-obj) &body) in CLISP. One can only do so in limited contexts, e.g. for calling recv() and be sure (not hope) never to GC before doing so.

Oh, BTW, the AFFI I designed in 1994/5 as an FFI for Amiga-CLISP passes
addresses to Lisp arrays. It has a i) moving GC but no ii) callbacks
nor signal handlers, thus no reliability problem (not considering
buffer overflows and heap corruption here :-) and thus fulfils contract 2).
I'm specifically not saying that the ability to pass address of Lisp
arrays to foreign code wouldn't be useful or speedy. But it also
has to be reliably defined.

Regards,
	Jörg Höhle.
who's working on SW-reliability and dependability at
Deutsche Telekom / T-Systems.