Re: Dylan 2016: Evolving Dylan

On Apr 18, 2014, at 2:25 AM, Bruce Hoult <[email protected]> wrote:

> I absolutely agree that UTF-16 as used by Windows, C#, Java is wrong. And that UTF-8 is best in files and  streams and in plain C programs.
> 
> What I don't see addressed there (or anywhere I've seen) is how one recognises a combining character or grapheme cluster.

You mean aside from the properties in the Unicode code tables?

> It seems to me that if such things can be recognised on decoding an input file/stream then a representation in which a character is the same size as a pointer allows complex beasts to be implemented as a pointer to a sequence of code points, interned and shared by all strings using them. Simpler characters (the vast vast majority) can be stored as their code point munged in the traditional dynamic language way to distinguish pointers from immediate values.

This is much like the idea I have, and have proposed before, for supporting Unicode text: <character> as a sequence of one or more <code-point>, <string> as a sequence of <character>. <Character> can then be an abstract (encodable) character or a grapheme cluster. (I can reply or CC the list with more detail, if you like.)