Some thoughts on Unicode.

Hi,=20

Firstly, I have strong feelings that, with any sort of comprehensive Unic=
ode
support, I should be able to open any arbitrary valid Unicode file, edit =
it,
and save it back to disk without losing data. This isn't the case at the
moment.

The FSF (well, version 21.3.1 of their Emacs) has split Unicode into rang=
es,
and created Mule character sets corresponding to those ranges. Any charac=
ters
outside those ranges are lost. (They seem to have some hack in place such=
 that
valid UTF-8 sequences are preserved on IO; this isn't the case for UTF-16=
,
though, so, for example, unknown characters in WinNT file names would bre=
ak,
if we were to implement this in our environment.)

Given current Mule, where characters are defined in terms of differing (=3D=
 9216
(* 96 96))-character sets, around twenty-three of which we would need to
define, to cover all of Unicode, never trash data, and keep the various H=
an
characters separate at the same time, it's not immediately obvious to me =
that
implementing support for all of Unicode in this architecture is a good id=
ea.=20
unicode_to_ichar in unicode.c would need huge tables to deal with everyth=
ing.

Beyond that, we can't move to pure Unicode as an internal representation,
because we need to keep Han characters de-unified.=20

(We could implement a stupid hack such that any unrecognised Unicode code=
point
that's read in, results in the associated buffer character having an inte=
ger
specifier associated with it, that records this code point, and have the
Unicode coding systems take note of those specifiers. But that would be s=
tupid
and hackish, and we don't want to do it.)

What I'm about to do, if no-one convinces me that it's a stupid idea, is =
go
and code up changes such that Ichars are almost UCS-4, and the arrangemen=
t of
Ibytes almost UTF-8.

By "almost UCS-4" I mean "use integer values above the maximum permitted =
UCS
code point to de-unify Han characters"--one block for Kanji, probably usi=
ng
jisx0208's ordering, one for Chinese Traditional (probably Big5; is CNS11=
643 a
better alternative there?), one for Chinese Simplified, probably using
GB2312's ordering, one for Korean (are there alternatives to ksc5601? My
understanding is that Han characters are relatively infrequently used in =
South
Korea, certainly compared to the situation in Japan and China, so there m=
ay
(thankfully:-) not be). My instinct is not to de-unify them beyond that--=
so,
adding jisx0203 only adds those characters it doesn't have in common with
jisx0208, adding one of [Big5, cns11643] only adds the characters not pre=
sent
in the other one. This mightn't be as fast for redisplay, but if we choos=
e the
character set properly, that's only going to be an issue for infrequently
used characters.=20

To explain "almost UTF-8," I'm going to have to digress a bit.=20
http://www.hackcraft.net/xmlUnicode/#sect422 gives a nice summary of UTF-=
8,
with a useful diagram. Now, the highest Unicode code point that Unicode c=
laim
will be allocated is #x10FFFF. This encodes in UTF-8 as "\xf4\x90\x80\x80=
";
four octets, first one #b11110100.

Algorithmically, you can determine the length of a UTF-8 character by sta=
rting
at the most significant bit and counting the number of bits set until the
first clear one. So, no valid Unicode code point has an encoding with mor=
e
than four of its bits set before you get to the first clear bit.=20

What I propose is that we say that any character in our Ibyte encoding th=
at
has its first five bits set is still four octets long, and its value afte=
r
masking should be added to 2^22 to get its XEmacs internal code.

This gives us 2^20 (2^21, if we want to make it _really_ unintuitive to p=
eople
used to more normal implementations of UTF-8) non-Unicode code points to =
play
with, with a hit of one octet per character in buffer representations for
current "Dimension-2 official" characters, but a saving of one per charac=
ter
for control characters, no hit for ASCII or anything below U+0800, and no=
 hit
for "Dimension-2 private" characters. The algorithmic complexity is sligh=
tly
worse, but isn't inordinate, and there are thoroughly optimised
implementations out there that we may be able to use (for the ibytes-to-i=
chars
and ichars-to-ibytes transformations.)

Other things that I'll have to change if I go and do this; the Unicode co=
ding
systems will have to be rewritten (easy), as will the other coding system=
s
(not so easy).

Any code that looks at charsets at the moment will have to be changed.=20
However, I am of the opinion that this code _should_ be using latin-unity=
 in
some shape or form at the moment, even if it isn't, and I also think that
latin-unity's API can be preserved in this new framework. So, if I port, =
e.g.=20
VM's and GNUS' code to use latin-unity a) it's cleaner, and better behave=
d in
an environment where multiple Roman character sets are being used, even i=
n
older XEmacs and b) it'll work in the new environment.

The tables that currently hold mappings from the Han character sets to Un=
icode
will need to be kept around, and which Han character set to use when look=
ing
at Unicode should be determined by the language environment, much as it i=
s at
the moment. (My intention is that the CJK Unicode code points are never u=
sed
internally, which is a waste, but it does underline that we're committed =
to
not unifying Han characters.)

The X11 faces code needs to, basically, do what non-XFT Mozilla did; keep
around mappings of X11 font registries to Unicode code-points. But, in
addition, we'll keep around information on the extra Han character sets w=
e're
using and what X11 registries they map to (there shouldn't be any need to
actually have per-character mappings, except perhaps for the characters b=
eyond
our chosen Japanese, Chinese Simplified, Chinese Traditional and Korean
character sets, and even there I think we may be able to use ranges a lot=
 of
the time.)

What's Windows and XFT doing at the moment with unified characters? As I
understand it, those APIs can't distinguish between CJK versions of han
characters, and what gets displayed depends on the fonts used. It may be =
worth
setting up something such that we _know_ that, eg, MS Mincho is Japanese
(hello, \ displayed as =A5 :-) and doing something based on that.

It may be worth preserving some of the charset API--though it can only be
broken for non-East-Asian character sets. Since the little application co=
de
that uses it does seem to me to be aimed at East-Asian users, that may be=
 a
win.=20

Bye, I hope not _everyone_ is on holiday, so that anything stupid about t=
his
can be made clear :-),

	- Aidan
--=20
Like the early Christians, Marx expected the millennium very soon; like
their successors, his have been disappointed--once more, the world has sh=
own
itself recalcitrant to a tidy formula embodying the hopes of some section=
 of
mankind. (Russell)