foreign strings and Unicode?

Hi,

I'd like to share ideas on how to deal with Unicode in foreign strings.

Lispworks does it, CMUCL doesn't have Unicode - others?

CLISP has Unicode and an FFI, but they are not fully integrated. To be more precise, CLISP's FFI can only make automatic use of "1:1" encodings, such as ASCII, ISO-8859-n etc. No UTF-8 (variable size) or UTF-16 (fixed 16bit size). The latter would have to be passed as pointers to arrays of uint8 and cannot be declared as C-STRING.


I *highly* welcome proposals for how an API which works with Unicode would look like.

What could a function declaration look like, which accepts (what format of) unicode strings?

I've been thinking about things like
(with-c-string (varname "foobar" &optional charset-or?-encoding)
  &body ...)
-> FOREIGN-VARIABLE of type (C-ARRAY uint8-or-character-or-what? <len>)
(FFI::FOREIGN-SIZE *) -> length in 8bit bytes
or something closer to make-array with &key :length :initial-contents ...
 + \0 at the end or not
 + what is the size, in what units?
 + read-only, empty buffer space or read-write with length possibly /= initial-contents

That would do to stack allocation of foreign strings what ext:convert-string-to-bytes does in Lisp.
CLISP's EXT:convert-string-from/to-bytes is at http://clisp.cons.org/impnotes/encoding.html#string-byte


UFFI has http://uffi.med-info.com/manual/r1286.htm
 convert-from-foreign-string foreign-string &key length null-terminated-p => string
but that's not enough to deal with various unicode formats.


BTW, http://uffi.med-info.com/manual/r1520.htm
find-foreign-library names directories & drive-letters types => path
   probably missing / bad conversion: ^^^ &key
DRIVE-LETTERS doesn't look portable at all.

Regards,
	Jörg Höhle.