Re: Migrating from UFFI to CFFI

On Thu, 17 Oct 2013 06:27:27 +0300, Kevin Rosenberg <kevin-HJRc7zDS/[email protected]> wrote:

> On Oct 16, 2013, at 11:07 AM, Victor <[email protected]> wrote:
>> ? (uffi:convert-from-foreign-string (uffi:convert-to-foreign-string "абв"))
>> "^Z^Z^Z"
>>
>> I suspect that the problem is in the uffi:convert-to-foreign-string function.
>
> Your suspicion may well be right. I look a look at the code and see that
> uffi::%convert-to-foreign-string for ccl stores two bytes of zeros after
> storing the bytes returned by ccl:encode-string-to-octets.
>
> On the reverse, uffi:convert-from-foreign-string uses the internal
> ccl::%get-utf-8-cstring for decoding :utf-8 encoded foreign strings.
>
> Perhaps something has changed in the internals of ccl utf-8 representation
> since I wrote, tested, and committed that code in Feb 2010. If you see some
> useful ideas for an update, please forward them to me and I'll be glad to
> commit a fix to work for UFFI to work with the current version of ccl.
> Ideally, there may be documented, exported functions for encoding/decoding
> internationalized strings in the current version of ccl.


Hi Kevin,

Indeed, OpenMCL-devel subscribers experienced the same problem. This
is a response that I've got from Sergey.

The mail from Sergey was Cc-ed to the openmcl-devel mailing list but
it did not appear there. I quote him below:

| 1) First of all I set ccl:*default-file-character-encoding* and
|    ccl:*default-socket-character-encoding* to :utf-8
|
| 2) And then I patched uffi soures in quicklisp directory

See these modifications as a patch attached to this message.

| Additionally i've defined encoding for still widely used cp1251, but
| I think it's offtopic.

I've applied these changes and indeed, Sqlite3 backend now is capable
of dealing with UTF-8 encoded strings. However, I also noticed that
the problem with UFFI:CONVERT-* functions still remain. Moreover, I've
also stuck into the problem that Sergey noticed some time before:

http://clozure.com/pipermail/openmcl-devel/2013-April/014220.html

it appears that the CCL::%GET-UTF-8-CSTRING function is not a reliable
one.

I think that if confirmed, this function has some internal problems
and should be either fixed or avoided.

The two zeroes in the end are a UTF-8 version of the 0 character
terminator that ordinary C strings have. CFFI relies on the babel
package to get its size, but I guess that this thing is static enough
to hard-code it for UTF-8.

Thanks!
Victor

p.s. CP1251 encoding is not defined by the patch but I believe it is
a good thing to do in the future.

_______________________________________________
CLSQL mailing list
[email protected]
http://lists.b9.com/cgi-bin/mailman/listinfo/clsql

diff --git a/src/i18n.lisp b/src/i18n.lisp
index a46b535..36577e5 100644
--- a/src/i18n.lisp
+++ b/src/i18n.lisp
@@ -202,5 +202,6 @@ foreign function."
 
   #-(or (and allegro ics) (and sbcl sb-unicode) (and clisp i18n))
   (length str)
-
+  #+(and openmcl openmcl-unicode-strings)
+  (ccl:string-size-in-octets str :external-format (ccl:make-external-format :character-encoding (or encoding *default-foreign-encoding* ccl:*default-file-character-encoding* :utf-8)))
 )
diff --git a/src/strings.lisp b/src/strings.lisp
index 430d8ab..1d2f875 100644
--- a/src/strings.lisp
+++ b/src/strings.lisp
@@ -84,7 +84,11 @@ that LW/CMU automatically converts strings from c-calls."
   (let ((stored-lisp-string (gensym)))
     `(let ((,stored-lisp-string ,lisp-string))
        (if (stringp ,stored-lisp-string)
-           (ccl:with-cstrs ((,cstring ,stored-lisp-string))
+           (ccl:with-encoded-cstrs
+               (or *default-foreign-encoding*
+                   ccl:*default-file-character-encoding*
+                   :utf-8)
+             ((,cstring ,stored-lisp-string))
              ,@body)
            (let ((,cstring +null-cstring-pointer+))
              ,@body))))