Re: wide character support

Gary Byers wrote:
> 
> On Tue, 1 Apr 2003, james anderson wrote:
> 
> ...
> 
> I admit that I prefer the notion of "widening" a single lisp char/string
> type to the notion of making char/string be union types and making all
> operations on strings have to handle both cases.  I think that it's hard
> to implement the latter scheme correctly and efficiently.

a situation with two runtime modes, one "wide", one "narrow" is what i was
referring to.

> 
> (It may be the case that OSX is admirably Unicode-aware at its lowest
> levels; it's also the case that the POSIX layer - which OpenMCL
> targets - deals almost exclusively in terms of 8-bit bytes.  A lot of
> OS-level code would need to change if OpenMCL used 16-bit
> characters/strings internally.

i am not an osx programer, but in order to understand what xml tools should be
targeting when decoding unicode-encoded data i waded through the
text-management documentation. it reads as if the internal representation is
utf-16, surrogates and all, with appropriate operators for positioning-in-string.

>                                That's not a compelling argument
> against that change, but it's a factor.  OpenMCL's stream buffering
> code exploits the fact that characters have the same size externally
> as they do internally; there are undoubtedly other places where code
> is currently able to exploit similar assumptions.)
> 
> There would need to be ways of specifying additional information about
> the :EXTERNAL-FORMAT of files.

integrating this with the standard stream interface is a separate question.

i cannot depend on stream support for external formats, as i am trying to stay
portable, so i do the utf-8/16/32 decoding myself. it's just that, in order to
support the base plane one needs 16 bits.

> 
> One source of concern (admittedly borne of ignorance) is that UTF16
> is [] a variable-width coding system.  My understanding is
> that the escaping features are relatively rarely used in practice,

instances have now appeared.

> but I don't know that we'd want SCHAR and AREF and ELT and LENGTH and
> ... to be aware of those features.  (Something, somewhere might want
> to view a 16-bit string that contained the codes #xD800 #xDC00 as
> consisting of a single "character", but I'd want (SCHAR that-string 0)
> to return the [high surrogate code value] #xD800.)

it depends on what you're going to do with it. one certainly wouldn't want to
delete the first of a surrogate from a sequence and leave the second one
behind. i also wonder if one componnent of a pair would have a meaning in a
collating sequence.

what would your opinion be of 24-bit characters? given that, one could
represent the scalar values (values which denote the abstract characters)
directly, without need for surrogate pairs.

...