Re: Some thoughts on Unicode.

 Ar an triochad=FA l=E1 de m=ED L=FAnasa, scr=EDobh Stephen J. Turnbull:=20

 > Been there, done that, got the T-shirt.  It's called UTF-2000
 > XEmacs/CHISE, and it's available from Kyoto University or
 > www.m17n.org/utf-2000, IIRC.  Your proposal, IIUC, is exactly what
 > they did at first, although at this point they use "real" Unicode
 > (UTF-8) internally and convert at redisplay.  We have efficient
 > chartables for Unicode that can convert to font indicies very quickly;
 > it's not a terrible hit.

My understanding (yes, I don't read Japanese, so it's probably flawed) of
what UTF-2000 did was that they were keeping aound _all_ the old-school M=
ule
charsets, as well as having Unicode. That's what made keeping around so m=
any
tables necessary, and what ballooned it to 30MB. But if we're not
de-unifying Han characters, that shouldn't be a huge deal.=20

 > Let's not go there.  Just go straight to UTF-32, and deal with space
 > efficiency later.  The structures for widechar representations are
 > already in place, but the code for widechar buffers still needs to be
 > written, I think.

What do you mean by "widechar representations" and "widechar buffers" the=
re?
Given that Ichars are already encoded as ints, and we don't support
sixteen-bit systems, the actual size of characters won't ever get any
bigger. (Yes, I know in theory they're nineteen bit units.)

Or do you mean having Ibytes sixteen bits wide? But Mule's fast enough
as-is, and if we are unifying Han, the maximum size of a UTF-8 encoded
string is four octets, as it is for current Ibytes.

 > In the next release, we do NOT want fonts to be determined by charset,
 > we want them determined by "culture" (probably =3D=3D language). =20

Okay. I had thought that the reason Mule originally separated its Han
charsets was discernment, and thinking things through; if it's
incompatibility, for the sake of it, then moving away is fine.=20

 > In most cases UTF-16 plus a global language environment will be
 > sufficient, which will actually be a saving over the Mule encoding for
 > Asians.  Connoisseurs and Buddhist scholars will need language
 > information attached to text, presumably to an extent.A
 >=20
 >     Aidan> However, I am of the opinion that this code _should_ be
 >     Aidan> using latin-unity in some shape or form at the moment,
 >=20
 > latin-unity is an egregious kludge; buffer representation should be
 > Unicode, and that automatically gives us proper identification of
 > Latin characters across ISO 8859 coded character sets. =20

I know it's a kludge--I still think its API should be preserved. An
efficient way to ask "can this buffer be encoded in iso-8859-1 without
losing data" would be, and is, worthwhile. I've a local modification to V=
M
that asks pretty much that, moving through a list of the preferred coding
systems while working out what MIME character set to use for a message. (=
So
if a message only contains Latin-1 it's sent as latin-1, if it contains
Latin-1 + the Euro sign it's sent as Latin-9, if it contains characters
outside of Latin-9 it's sent as UTF-8.)

 > It will cause issues because of Han unification, but we can deal with =
Han
 > disunity later.  The charset coverage check should be embedded in the
 > coding system I/O routines, I'm not sure where.

Oh, okay, so when you write a new coding system you provide a function th=
at
checks whether a region can be encoded using that coding system, or one i=
s
generated automatically. (Or there's a flag that's passed to the coding
routines that says "flag errors.")

 > An alternative experiment:

(Really, at this point, in August 2004, we should be past experimenting :=
-()

 > What I've been doing (15 minutes at a time :( ) is converting the
 > buffer representation to UTF-8 using Mule character codes.  Ie, Ichars
 > are the same, but instead of using Mule-style leading bytes, the
 > buffer representation does int-to-utf8 on the Ichar. =20

How does that perform with iso-2022-based coding systems? Is there much o=
f a
hit? I suppose, though, that's not going to be done that often, compared =
to
converting to UCS-4 for redisplay.=20

I'd like to get my hands on that code if that's at all possible. (I'm qui=
te
certain I have more free time to hack at it than you do :-)

 > The next step after that is to change the internal representation to
 > UTF-8 and use unicode-to-char to index into the fonts (except for Xft).

? Oh, as in, we can call XftDrawStringUtf8, after checking whether a give=
n
code point is available in the font.=20

 > Finally we can cache font indicies in a Unicode chartable.
 >
 > The next step after that is to arrange for the codecs to set extents
 > in the buffer for the charsets decoded. =20

What does that give us? Except, perhaps, caching information for writing
things out again. Hmm.=20

--=20
Like the early Christians, Marx expected the millennium very soon; like
their successors, his have been disappointed--once more, the world has sh=
own
itself recalcitrant to a tidy formula embodying the hopes of some section=
 of
mankind. (Russell)