Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

Hmmm...

Perusing the TRE specification shows that it can be compiled to
allow both "wchar" characters (which I assume means strings of
type "wchar_t"), as well as"multibyte" characters (which I assume
means variable-length characters, for example UTF-8).  The
two options are separate.

If this is not the case, then Ville, please correct me.

Here's why:

1) I sometimes get UTF-8 (variable-length character) email.  Some of
   it is even useful (i.e. a message that has some Japanese in it).
   If we go to UTF-32, we need to translate correctly between UTF-8
   and UTF-32 both inbound and outbound.

2) On the other hand, I have _never_ gotten wchar email; I *think*
   wchar email would display with a lot of little splats for each
   of the zero-extended bytes (which I do see on some genned files
   when I screw up).

3) TRE does not care about wchar_t vs. multibyte, so that doesn't matter.

4) However, UTF-16, UTF-32, and wchar_t are machine-endian dependent
   and so should use a BOM (Byte Order Mark) preamble, but
   unfortunately the BOM preamble in a text file screws up the shebang
   magic number scheme.  On the other hand, UTF-8 doesn't have a BOM
   and so doesn't care.

5) Python uses UTF-32 (32 bits per character, 4 bytes, aka UCS-4) as
   the internal representation and Python seems to work just fine.

6) But going to 32 bits/character is going to wreak havoc on "embedded
   system" CRM114.  All data storage space will grow by 4x including
   the data window and the isolation window.  Some of the classifier
   statistics files will also grow 4x (Compressive and Correlative
   will, for instance).

7) For that matter, it will be a PITA for everything that does I/O
   because we'll need to allocate a translation buffer, translate
   to/from UTF-32 inside to UTF-8 externally, and then actually do the
   I/O. ... but we can't do it to every file, because some files like
   statistics files currently have mixed ASCII and binary components.

8) Tokenizing classifiers like OSB, Markov, Hyperspace, etc. operating
   on UTF-32/wchar will see an input stream dominated by zeroes, and
   so the hashing system will need to be robust against zero-dominated
   streams (doable, maybe a little slow and tricky and definitely not
   upward-compatible from what we have now).

9) Non-tokenizing classifiers (i.e. Bit-entropy, Compressive,
   Correlative, String-Kernel, etc) classifiers will have major
   performance and accuracy hits in a UTF-32/wchar world, because the
   average LATIN-1 document will become 3/4 NULL bytes and (on the
   average) 7/8 zero bits rather than the almost-balanced ratio we
   have now.  At least for bit entropy, simply growing the file 4x
   will not recover the accuracy.  [[ Note that we have found that the
   bytewise string kernel used on Chinese actually filters spam pretty
   well! ]]  This performance and accuracy hit will be intense.
 
Yes, I now have a headache. 

   - Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/