Re: wide character support

On Tue, 1 Apr 2003, james anderson wrote:

>
> i could understand that that might have made sense for the original target
> domain, but is it that hard to have alternative builds which use 16 bit
> characters. i'm not even arguing for an imlementation which handles two
> classes of string. given, for example, that your base os calls are now in
> terms of utf-16 encoded strings, what is the argument for 8-bit lisp strings?
>

First of all: sorry to have missed the original message.  I was out of town
for a few days.

I admit that I prefer the notion of "widening" a single lisp char/string
type to the notion of making char/string be union types and making all
operations on strings have to handle both cases.  I think that it's hard
to implement the latter scheme correctly and efficiently.

(It may be the case that OSX is admirably Unicode-aware at its lowest
levels; it's also the case that the POSIX layer - which OpenMCL
targets - deals almost exclusively in terms of 8-bit bytes.  A lot of
OS-level code would need to change if OpenMCL used 16-bit
characters/strings internally.  That's not a compelling argument
against that change, but it's a factor.  OpenMCL's stream buffering
code exploits the fact that characters have the same size externally
as they do internally; there are undoubtedly other places where code
is currently able to exploit similar assumptions.)

There would need to be ways of specifying additional information about
the :EXTERNAL-FORMAT of files.

One source of concern (admittedly borne of ignorance) is that UTF16
is (technically) a variable-width coding system.  My understanding is
that the escaping features are relatively rarely used in practice,
but I don't know that we'd want SCHAR and AREF and ELT and LENGTH and
... to be aware of those features.  (Something, somewhere might want
to view a 16-bit string that contained the codes #xD800 #xDC00 as
consisting of a single "character", but I'd want (SCHAR that-string 0)
to return the bogus (?) character whose code is #xD800.)