Re: Some thoughts on Unicode.

>>>>> "Aidan" == Aidan Kehoe <[email protected]> writes:

    Aidan> What I'm about to do, if no-one convinces me that it's a
    Aidan> stupid idea, is go and code up changes such that Ichars are
    Aidan> almost UCS-4, and the arrangement of Ibytes almost UTF-8.

Been there, done that, got the T-shirt.  It's called UTF-2000
XEmacs/CHISE, and it's available from Kyoto University or
www.m17n.org/utf-2000, IIRC.  Your proposal, IIUC, is exactly what
they did at first, although at this point they use "real" Unicode
(UTF-8) internally and convert at redisplay.  We have efficient
chartables for Unicode that can convert to font indicies very quickly;
it's not a terrible hit.

    Aidan> What I propose is that we say that any character in our
    Aidan> Ibyte encoding that has its first five bits set is still
    Aidan> four octets long, and its value after masking should be
    Aidan> added to 2^22 to get its XEmacs internal code.

Let's not go there.  Just go straight to UTF-32, and deal with space
efficiency later.  The structures for widechar representations are
already in place, but the code for widechar buffers still needs to be
written, I think.

In the next release, we do NOT want fonts to be determined by charset,
we want them determined by "culture" (probably == language).  In most
cases UTF-16 plus a global language environment will be sufficient,
which will actually be a saving over the Mule encoding for Asians.
Connoisseurs and Buddhist scholars will need language information
attached to text, presumably to an extent.

    Aidan> However, I am of the opinion that this code _should_ be
    Aidan> using latin-unity in some shape or form at the moment,

latin-unity is an egregious kludge; buffer representation should be
Unicode, and that automatically gives us proper identification of
Latin characters across ISO 8859 coded character sets.  It will cause
issues because of Han unification, but we can deal with Han disunity
later.  The charset coverage check should be embedded in the coding
system I/O routines, I'm not sure where.

    Aidan> What's Windows

It unifies them, depending on font selection to disambiguate.

    Aidan> and XFT doing at the moment with unified characters?

Being very broken AFAIK.  Just plain doesn't work in XEmacs.  Maybe
the Tuebingen guys have something by now.

    Aidan> As I understand it, those APIs can't distinguish between
    Aidan> CJK versions of han characters, and what gets displayed
    Aidan> depends on the fonts used.

Mr. Ohta[1] notwithstanding, that is what should happen.  All of the
Japanese and Chinese national bodies have made far more egregious
identifications in their national standards than those Unicode's "Han
unification" makes.  They are the same characters (read any Japanese
textbook on Chinese poetry; of course they use Japanese fonts!)  What
differs is how the different languages like them to look.

An alternative experiment:

What I've been doing (15 minutes at a time :( ) is converting the
buffer representation to UTF-8 using Mule character codes.  Ie, Ichars
are the same, but instead of using Mule-style leading bytes, the
buffer representation does int-to-utf8 on the Ichar.  The next step
after that is to change the internal representation to UTF-8 and use
unicode-to-char to index into the fonts (except for Xft).  Finally we
can cache font indicies in a Unicode chartable.

The next step after that is to arrange for the codecs to set extents
in the buffer for the charsets decoded.  This may require a fairly big
rewrite of the current codecs because they take responsibility for a
whole stream, and therefore they may not be amenable to stopping,
creating an extent, and going on, but maybe it's not a big deal.

Footnotes: 
[1]  Masataka Ohta wrote an anti-Unicode polemic entitled "The
Japanese Language Is Now Endangered!"

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.