Re: Characters lost when saving UTF-8 file on win32 (21.5.16)

>>>>> "Vladimir" == Vladimir Weinstein <[email protected]> writes:

    Vladimir> I'm a bit confused now - I was under impression that
    Vladimir> xemacs 21.5 uses Unicode internally.

Not yet.  Here's the plan:

0.  Current Mule (ie, 21.4) has an explicit component of each
    character to indicate the charset, which is trivially translated
    to an X11 registry, and used to look up the font.  There is no
    internal concept of Unicode.
1.  Add functions to parse Unicode Consortium translation tables, and
    convert to internal tables.  Provide unicode coding-systems which
    use these tables to translate to Mule code internally.  Provide
    unicode<->Mule code functions for individual characters (unicodes
    are not of character type, they are integers).
2.  Change internal code to Unicode.  Provide unicode->X11 registry
    mapping tables for redisplay on X11.  Maybe swap representations
    of integers and characters (currently integers are 31-bit,
    characters are 30-bit with several bits unused).  This swap is
    trivial, but was vetoed 3 years ago on the grounds that we didn't
    support Unicode, but people do edit >1GB files (really!)

Currently we have completed stage 1.

The problem of not corrupting data is independent of the Unicode
transition, and we *absolutely* have to do something about that soon
(and I hope to get it backported to 21.4, too---it's a historic
problem with Mule, nothing new).

    Vladimir> The characters that do get whacked are from different
    Vladimir> parts of Unicode - circled Kanas,

The circled kana are corporate characters not part of JIS X 0201,
0208, or 0212, thus will not be supported by existing Mule charsets.
There is no Devanagari charset in XEmacs.  I would guess that the
missing Hangul are similar to the circled kana, not part of the
relevant standard.  (I suspect that XEmacs Mule follows the JIS X
0208.1990, JIS X 0212.1990, and KSC 5601.1991 standards).

    Vladimir> The problem of a good Unicode editor is very much
    Vladimir> present. It would be great if xemacs could fill that
    Vladimir> space :)

That's where I'd like to go!  But unless we get more people who are
willing to work on XEmacs in some way[1], it's going to take a fair
amount of time.


Footnotes: 
[1]  Doesn't have to be directly on that---it's possible to "trade"
among developers, or reduce the bug fixing burden on core developers,
etc, and so allow them to work on the features you want.

-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.