trying to add interoperable foreign string support

Hi,

Matthew Danish said:
>UFFI does not currently support:
>foreign funcall of function pointers (or function pointer types, I
>think), out and in-out function arguments, casting, and callbacks.
Tim Bradshaw misses (in comp.lang.lisp) most: callbacks and string encodings. Different priorities, obviously depending on needs.

I'd like to suggest for inclusion in UFFI something along the following for dealing with encodings and foreign strings, which I've had plans to include in CLISP, modeled after Lispworks:


(with-foreign-string (fvar e-var b-var
                      :encoding (ext:make-encoding :charset 'charset:utf-16)
                      :null-terminated-p (if a b c))
    "abc"
  (values fv e b)) ; examples of use at bottom

;-> FV: foreign variable/object, can be passed to foreign function
; E variable holding element count (including terminating zero if present)
; B variable holding bytecount

The key is the :encoding keyword (:external-format in LispWorks). In CLISP it can be an arbitrary encoding, e.g. charset:ascii, charset:utf-32 (CLISP knows ~120). In Lispworks, it seems it can be only :unicode, :ascii or some MS-Windows multi-byte code page. In implementations which don't have encoding, this would be ignored.

The problem which prevented me so far from implementing this is :null-terminated-p.
The implementation needs to know the number of terminating zero bytes. This depends on the encoding. If Lispworks only knows :unicode (as synonym for 2 bytes) and the rest is based on extensions of ASCII, like iso-8859-1, then it's easy for Lispworks. In CLISP, it's not so, as the number of zeroes may vary from 1 (iso-8859-1 to 4 (unicode32-big-endian) - how to know (need additional support function, e.g. encoding-zeroes)?


;; LispWorks has WITH-FOREIGN-STRING (fv element-count byte-count :external-format :null-terminated-p) string body

;; This is better than [CLISP's] ffi:c-string because there's no 1:1 limit like on [CLISP] custom:*foreign-encoding*,
;; so Unicode strings can be converted.
(defmacro with-foreign-string ((foreign-variable element-count byte-count
                                  &key (encoding 'custom:*foreign-encoding*) (null-terminated-p 'T))
			       string &body body)
  `(ffi::exec-with-foreign-string
    (lambda (,foreign-variable ,element-count ,byte-count) ,@body)
    ,string ,encoding ,null-terminated-p))

(defun (setf window-title) (handle title)
  (with-foreign-string (fstring element-count byte-count
       :encoding #<any implementation-dependent encoding, e.g. UTF-16>
       :null-terminated-p t)
      title ; given as Lisp-string
    (declare (ignore byte-count element-count))
    (c-api-set-title fstring)))

(defun length-in-bytes-using-encoding (string encoding)
  (with-foreign-string (fstring element-count byte-count
       :encoding encoding)
      string
    (declare (ignore fstring element-count))
    byte-count))) ; feeling overkill

There would also be convert-from-foreign-string, supporting an encoding/external format keyword.

Any comments on how useful or interoperable this may be?

Regards,
	Jorg Hohle.