Re: character encoding problem

Pascal Bourguignon writes:

 > >> utf-8 is able to encode all the codepoints between 0 and
 > >> 1,112,063, but if you encode an invalid codepoint, you should
 > >> not be surprised if a system cannot convert it into a unicode
 > >> character!

I'm having trouble making sense of this.  By encode you mean perhaps
translate that number to a utf-encoding, such as #xd83c => 237 160 188
So utf-8 defines a mapping with the large domain above (which includes
many non unicode code points) to byte sequences.  Does it make sense to
say that utf-8 defines a mapping from the non-code-points or only the
code points in that range?

 > Also, you should probably use an encoding with a input-error substitution:

I guess so, but then I lose information that might be valuable.

 > (code-char #xd83c) should return NIL.

But code-char is not tied to utf-8.  Lisp characters evidently don't have
to be unicode characters.

 > It does not, which is not consistent with convert-string-from-bytes.

But that function requires an encoding argument.  Perhaps number IS the
code point for a character in another encoding.  

 > On the other hand, utf-8 can encode and decode any integer between
 > 0 and 1112063, so the conversion of the code point between the
 > utf-8 octet sequence is possible and valid in both directions.

I can't figure out how that statement is consistent with the failure
on #xd83c, which is certainly in that range.  Encode means generate
corresponding bytes, like convert-string-to-bytes does?  But does it
make sense to say that utf-8 decodes an integer?  Would that be the
integer interpretation of the byte sequence?  And evidently it does
not decode the integer interpretation of this particular byte sequence.


_______________________________________________
clisp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/clisp-list