Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

On Tue, Sep 23, 2008 at 8:46 PM, Bill Yerazunis <[email protected]> wrote:
> Perusing the TRE specification shows that it can be compiled to
> allow both "wchar" characters (which I assume means strings of
> type "wchar_t"), as well as"multibyte" characters (which I assume
> means variable-length characters, for example UTF-8).  The
> two options are separate.
>
> If this is not the case, then Ville, please correct me.
[...]

> 1) I sometimes get UTF-8 (variable-length character) email.  Some of
>   it is even useful (i.e. a message that has some Japanese in it).
>   If we go to UTF-32, we need to translate correctly between UTF-8
>   and UTF-32 both inbound and outbound.

In a wchar_t-based application: Check!  (Though you might check those
email headers again for a 'charset' spec; see further below for a
sample)

If we just take the aspirin and say: we SUPPORT UTF-8 ONLY _and_ make
absolutely sure TRE actually fully copes with Unicode through the MBCS
interface, then no expanding/compression of input data is needed,
resulting in the new situation being exactly like it is in todays
CRM114. Less CPU load, lowest possible memory footprint.

> 2) On the other hand, I have _never_ gotten wchar email; I *think*
>   wchar email would display with a lot of little splats for each
>   of the zero-extended bytes (which I do see on some genned files
>   when I screw up).

wchar_t is for /internal representation only/. For UTF-x storage
formats, you have BOMs to guide the encode/decode bits of UTF-aware
software while [other] 'character encodings' are marked using headers
or other bits 'n pieces (In the case of email: see your email headers:
'charset' encoding -- a sample:
----
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: 8bit
----
).
Hence you can never receive wchar_t email as wchar_t is not an
encoding but an internal storage format. You *might* load UTF-16
encoded files into CRM114 though; however UTF-16 encoded email would
be illegal if I still got my [email] RFCs right.

To prevent lots of headaches, I think it's best to stick to UTF-8
without BOM exclusively. That makes matters easy as then the only
place we actually have to recognize *characters* instead of bytes is
the regex section, as feature hashing is byte oriented: the
'tokenization' done before the hashing is performed through the
regexes -- mind though that the 'isgraph()' speedup code in several
classifiers has to go though: /everything/ has to be processed through
TRE regexes and no speedup shortcuts for special regexes like
'[[:graph:]]+' any more!


> 3) TRE does not care about wchar_t vs. multibyte, so that doesn't matter.

I never tested if TRE handles Unicode regexes or content when passed
to it through the MBCS/char interface. (e.g. regexes including
U+xxxxxx characters)
Just to cut down on the Unicode discussion - which indeed is a
headache - I think a specific test for that is in order.


> 4) However, UTF-16, UTF-32, and wchar_t are machine-endian dependent
>   and so should use a BOM (Byte Order Mark) preamble, but
>   unfortunately the BOM preamble in a text file screws up the shebang
>   magic number scheme.  On the other hand, UTF-8 doesn't have a BOM
>   and so doesn't care.

Welllll... UTF-8 /may/ have a BOM and it happens a lot in the wild for
files, but it's used as an encoding tag rather than endianess
indicator.

For reference: see 2nd Q in the UTF-8 FAQ section here:
http://unicode.org/faq/utf_bom.html#UTF8


> 5) Python uses UTF-32 (32 bits per character, 4 bytes, aka UCS-4) as
>   the internal representation and Python seems to work just fine.

Yup. It expands bytes into ints and compacts them into UTF-8 on output again.

I never said it wouldn't work ;-) - though it adds increased memory
footprint and CPU load compared to the current situation of CRM114.

> 6) But going to 32 bits/character is going to wreak havoc on "embedded
>   system" CRM114.  All data storage space will grow by 4x including
>   the data window and the isolation window.  Some of the classifier
>   statistics files will also grow 4x (Compressive and Correlative
>   will, for instance).

So: shall we stick with UTF-8 throughout the pipeline? You've got my
vote on that one, because it is a great excuse to kick all
encoding/decoding issues out the door and leave it for external apps
to resolve.

That only leaves RAW byte inputs hanging [*], but I can cope with that.

[*] TRE might view those raw bytes as MBCS elements and mix them into
characters while they aren't. When we say CRM114 does support UTF-8
only, RAW is out as well.


By the way: note that 'MBCS' is something entirely different from
UTF-8; hence my current 'I believe it when I see it' attitude towards
TRE and Unicode support through the char/MBCS interface.
UTF-8 is just /a/ MBCS encoding.

I did a google looking for a good explanation; didn't find what I was
looking for but this has to do:
  http://www.ibm.com/developerworks/linux/library/l-benson.html
Pay particular attention to the "What if you can't use UCS-2?" section
while ignoring the fact that UCS-2 is outdated; that's not the point
there.



> 7) For that matter, it will be a PITA for everything that does I/O
[..]
> 8) Tokenizing classifiers like OSB, Markov, Hyperspace, etc. operating
[...]
> 9) Non-tokenizing classifiers (i.e. Bit-entropy, Compressive,
[...]

Yup. Yup. And yup. Hadn't thought of #8 and #9 there; good you mention
them as they can have some serious impact.


> Yes, I now have a headache.

Suggested Aspirin: stick with UTF-8 throughout the pipeline and stick
with it hard: no other 'encoding'/format/whatever allowed anywhere.
When people want to do 'encodings' and such, they'll have to do it
outside.

That means: no conversion to wchar_t and (depending on how you feel)
all data stored in bytes as 'char' (or for the nitpickers it's the
'unsigned char' or even more anal: uint8_t --> use 'char' to stay away
from a whole slew of compiler warnings when forwarding input to
fopen(), etc. RTL functions, including str*() functions - which can
handle UTF-8 if you're careful not to strchr() for 'high-ASCII'
characters (i.e. chars with bit 7 set) and such).




-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/