Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

On Wed, Sep 24, 2008 at 8:20 AM, Eugene Crosser <[email protected]> wrote:
> In mail classification, there *has* to be multiple charset support.

I know. I am only saying those encoding should be handled outside of
CRM114 - as you seem to say as well, though you seem to expect it in
the 'wrapping application' with which you /might/ mean crm114? - ,
because 'including it' would mean incorporating full locale/iconv/etc.
support and that's a hassle we can do without. As a feature, it'd be
'nice', since a lot of CRM114s (?) are processing email, but I don't
know if Bill would like adding libiconv to the CRM114 dependencies +
adding a 'encode/decode', i,.e. codepage 'transform' script command to
handle this kind of thing inside CRM114 (contrasting sycall /iconv(3)/
as one would have to do today).


What I was looking at though is not so much moving [lib]crm114 to
wchar_t - as Bill mentioned as well: there are a few issues with that!
- but keeping it at the current byte datasize, while supporting
Unicode through the use of UTF-8 encoding throughout.

In short: can we have Unicode without wchar_t as 'internal representation type'?


Theoretically: yes.

In practice: <waves hands />

Right now, effectively we (= crm114) have 'system/RTL default' MBCS
support in our regexes; which at least partly explains why I can't get
TRE to accept Unicode characters when using the [currently used]
char-based API.


Elaboration:

Tests and code review of TRE show that its 'char' sized interfaces
only support MBCS (mb*() RTL functions), which are 'C' locale specific
and NOT UTF-8, at least not on my systems.
(See also the TRE code: look for regcomp/regexec/regwcomp/regwexec
wrappers: regcomp/exec call the core in STR_MBS 'mode', while
regwcomp/wexec use the STR_WIDE 'mode' - which is what we're after
when talking 'Unicode' for all systems (including the ones without
'UTF8' locales).

Recent Linuxes/UNIXes at least seem to support UTF-8 locales by using,
for instance, "en_US.UTF-8", for setlocale() -- a call which must be
added to the crm114 init code to make it work -- but then again,
Windows (and probably quite a few older/non-Linux kernels) don't
support UTF-8 based MBCS locales: quoting from the M$ documentation
for that platform series:

----setlocale()---
The set of available languages, country/region codes, and code pages
includes all those supported by the Win32 NLS API except code pages
that require more than two bytes per character, such as UTF-7 and
UTF-8. If you provide a code page like UTF-7 or UTF-8, setlocale will
fail, returning NULL. The set of language and country/region codes
supported by setlocale is listed in Language and Country/Region
Strings.
----

which adds a bit of a hassle at least for the Windows port as that
means I have to add libutf8 to allow for proper UTF-8 support and
probably a few tweaks 'n tugs to kill off the system-provided mb*()
RTL functions and have those replaced by the libutf8 ones. Not a
problem per se, but definitely a point to watch.


So when we want to stay with least sized mem footprint with
char/byte-based MBCS strings in a UTF-8 locale (which isn't the case
for today's CRM114 due to locale issues), we'll have to compile crm114
+ libcrm114 + libTRE on top of libutf8 for all systems which don't
support UTF-8 MBCS locales - which will very probably include several
UNIXes.



URLs:

libutf8:
http://www.haible.de/bruno/packages-libutf8.html

to get a whiff of the locale hassle with M$ platforms (and elsewhere);
this is for C++, but it's the same for 'C', just different API:
http://bytes.com/forum/thread639483.html


The alternative is moving it all to wchar_t, which adds a few
drawbacks of it's own (footprint, overhead, increased CSS disc
requirements, hash [mal]functions, Windows @ 16-bit instead of reqd. ~
20 bits for Unicode v5.x). It's a matter of picking your poison.

Now that I've looked at it a while longer, I'd rather take the UTF-8
poison road; in a way it's probably easier for Bill as well - UNTIL he
steps out of the code and into the make/build environment, as I expect
a few 'surprises' in UNIX Land as well (while we'll certainly hit
others when we take the wchar_t road) -- which can be taken care of by
a good ./configure ;-)



(Note: hough CRM114 seems to work on 'characters', it mostly works on
'words' (features) as almost everyone (?) is using CRM114 for
classification. As such, it is not a pure 'character' but rather a
'byte'-*data*-based processing unit (despite several regex-based
character-oriented script commands) and one might use that to argue
that a move to wchar_t may not be the obvious road after all. Though
this reasoning can be considered more like an 'excuse' than anything
else. ;-) )





-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/