Re: BeOS port news and public patch

(sorry if I sent it twice, not sure since this *$% webmail unlogged me)

Selon "Stephen J. Turnbull" <[email protected]>:

> Please stop posting to multiple lists.  That's just going to get your
> posts lost and disconnected from their threads---many of us suppress
> duplicates.  Send patches you think are ready for inclusion in the
> development tree to xemacs-patches.  Send all other discussion to
> xemacs-design.
Sorry for that.
The first post had a patch so I crossposted, and someone told me 
xemacs-design might not be very active.
Then you replied to all, and I didn't bother changing the cc.
I used to not reply-to-all, but the current trend of mailing 
list to not set the reply-to to the list really bothers me.

> 
> >>>>> "François" == François Revol <[email protected]> writes:
> 
>     François> Maybe a "virtual" utf-8 charset would do, but it seems
> 
> UTF-8 can't possibly be a charset.  It is a specific transformation
> format for Unicode, but all Unicode variants share the same charset.
Yes I know that, however in some places the distinction of charsets 
from encodings is a bit blured...

> 
> Unicode is not treated as a Mule charset in XEmacs; it has special
> status because (in general) a Unicode code point may be 31 bits (cf
> the UTF-32 transformation format and ISO 10646), and Emacs characters
> only have 30 (and a large block of them is unavailable since they are
> already used for other character sets).
Argh, right, 2 bits are used by the list object to tell the type.

> So Unicodes are represented as integers, and all positive integers are
> Unicodes.  (In principle; of course we have translation tables you can
> use to discover whether an integer actually has a definition in our
> version of Unicode, see etc/unicode for the tables we use; they can be
> trivially updated from www.unicode.org.)  Nor do we directly output
> Unicodes to the GUI.  For one thing, this is very poorly documented
> except for the Utf8* extensions in XFree86 (and I assume the Unicode
> APIs for recent Win32 have decent docs from MSDN, but I don't do
> Windows).  For another, since there are very few reasonably complete
Neither do I :)
Actually the Unicode APIs in win are just renamed with _w appended 
to the name AFAIK, windows uses that ugly wchar stuff, nothing much
to document I think.

> free Unicode fonts, we need to support "filling in" from legacy
> character sets, but even XFree86's Utf8* extensions don't support
> that.  So instead we translate to legacy character sets, and there's
> no need for a "Unicode charset".
Granted, same for BeOS R5, which doesn't support Unicode overlays,
unlike R5.1 and Zeta, which do support them, and it works quite well.
It just uses a predefined font (it's customizable) for unicode planes 
not supported by the selected font.
I believe FreeType supports that now too.

> 
> However, there are a whole set of Unicode coding systems, which are
> used to translate blocks of text from one encoding to another.  See
> src/mule-coding.c for more information on those.
Yep I saw that file but didn't spend much on it yet.

> 
> You really need to get a clue about this stuff if you want to work
> with X?Emacs; for better or worse, Mule is based on ISO 2022 and that
> is not going to change without a large amount of work, which is barely
> begun yet.  If you create a variant of XEmacs which only works with
> UTF-8, even if that's only on BeOS, it will not get in to the mainline
> CVS tree.  You need to work with the existing framework.
I don't really want to fork, since it's likely not to be maintained 
at all if not by me.
The problem is the BFont class does have a method to set the 
font encoding, but the available list is quite limited:
B_UNICODE_UTF8    = 0,
B_ISO_8859_1      = 1,
...
B_ISO_8859_10     = 10,
B_MACINTOSH_ROMAN = 11
(note the ambiguity on encoding/charset)

My point was to try getting XEmacs to give my UTF-8 directly, 
since it already uses multibyte internally, instead of getting 
all those charsets and having to convert all them back to utf-8 
for display, taken that the redisplay code is already quite slow.
(it's not done from the window thread which is where it's 
supposed to be done, so I have to lock it quite often, that is in
every redisplay method. I tried to use the begin/end_frame/window
methods, but they don't seem to work the way I need, as other 
calls are sometimes called outside of the start/end pair :-()
At least in Zeta it seems much faster, and I suppose it's because 
of app_server double buffering the display)

> 
> You can download a copy of ISO 2022 from ECMA's website (I think it's
> www.ecma.ch) as ECMA-35.  I believe the Unicode standard is online at
> www.unicode.org, but ISTR it's (deliberately) hard to download the
> whole thing.
> 
> For Emacs-specific stuff, there are long comments in most of the files
> related to Mule in src/, in particular look in mule-charset.*,
> charset.h, mule-coding.*, file-coding.*, text.*, and unicode.c.  There
Yes I had a quick look to that, but it's a huge amount of stuff to 
digest:)

> is a fair amount of information in the Lisp Reference Manual and the
> Internals Manual (this is pretty sketchy, unfortunately) available
> from the Info directory node.
Yes I already read the info nodes about that, but I'll need a second 
pass I think. :)

>     François> Emacs has a very specific idea about what charsets are
>     François> (256 or 256^2 ?).
> 
> Try 94^n or 96^n.  UTF-2000 (see below) has extensions for 256 and
> 256^n charsets and I think some of it was in 21.2.x before UTF-2000
> forked from the XEmacs mainline, but if any of that is left in XEmacs
> 21.4 or 21.5 it's dead code.
Ok, will have a look.

> 
>     François> Has any of those attempts been published somewhere ?
> 
> There is a working version of XEmacs based on XEmacs 21.4 called
> XEmacs/UTF-2000 or CHISE.  See http://www.m17n.org/utf-2000, I think
> (it's all in Japanese, but you can download from there, still).
> XEmacs/UTF-2000 uses UTF-8 internally but translates from Unicode to
> legacy charsets, and then outputs using corresponding fonts AFAIK.
Ok, worth a look, thx for the link.

> The various patches for Xft have all been posted to xemacs-patches;
> search the archives for "xft".  There's link to the archives at
> http://www.xemacs.org/Lists/#xemacs-patches.  The Windows code is in
> CVS for ages; look at the files named src/something-msw.*.
As long as I can get any form of Unicode I don't mind, even if it's
UCS16 like windoze. I think it's much faster to convert between 
Unicode encodings than it is to convert between all those charsets.
(or maybe not ?)

> 
> You're working with 21.5, current CVS, right?  If not, get it and port
> your changes to it; your code will probably never get into 21.4.
Yes, I use cvs HEAD and regularily make a patch and reapply it to 
a fresh cvs. I should better cvs up on the modified version probably.

> 
> If you are already working on 21.5, have you bothered to list the src/
> and lisp/ directories yet?  If you do, you should immediately find
> unicode.c and unicode.el.  The first contains the utilities needed for
> translating Emacs characters to Unicode code points.  The second turns
> out to be irrelevant to your current purposes.
Yes, it's just that I hoped I could have not the need for that, 
but it seems unlikely.

> 
> I suspect you may have to steal int-to-utf8 code from somewhere; there
> is such code in mule-charsets.c, IIRC.  Please try to make sure that
> any such utility functions end up in unicode.c, and are not duplicated
> in several places.

Thanks for the infos,
François.