Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

On Tue, Sep 23, 2008 at 5:01 PM, Bill Yerazunis <[email protected]> wrote:
>
>   Would it be most logical if the C library API *always* used
>   wchar_t *text ? The wrapper application would be responsible for
>   translating whichever original encoding (including ASCII) into wchar_t
>   buffer.
>
>   Just an idea...
>
> Sadly, wchar_t is itself not consistent.  It's 32 bits on GCC (confirmed)
> and supposedly 16 bits on Windows.  Nominally, it's done by zero-extending
> LATIN-1 but the lack of portability is kinda damning...

Yup. Windows was an 'early adopter' there and matched wchar_t to the
16-bit Unicode space of Unicode standards up to and including v3.0
IIRC.
gcc is more hip and thus uses 32-bit ints to store unicode characters,
allowing for sufficient room for any character listed in Unicode v5.x.


Since you are descending into Character Mapping/Conversion Hell
anyway, I'd say screw Windows and go for the
1-full-integer-per-character plan a la gcc. The Windows CRM114 code
hackers (that may be me, I fear) will have to manage and support the
limited set there. Heck, I'll surely be able to invent some
preprocessor-assisted wickedness, no worries. Done that, been there.

[Checked my memory; not too bad.
http://en.wikipedia.org/wiki/Unicode
see 'Operating Systems' section there.
To help decide on what to do, I suggest you also read the section
'Unicode Transformation Format and Universal Character Set',
particularly the bits about UTF-32. Best practices and all that... In
this particular case I strongly suggest giving Windows the birdy.
It'll mean significant porting effort, but the end result is an easier
life for all.]




But before we decide on this stuff for real, I suggest we look into
TRE and see how well that one is handling Unicode plus a bit of CRM114
cost analysis:

1- Unicode int vs. ASCII byte is memory footprint x 4; for UTF-8
encoded content the multiplier would of course be less, but still > 2
for all but the most academic use. Plenty RAM these days, but x4 also
means slight performance hit as CPU cache misses will increase when
using the pattern matching/transforming l33t skillz of crm114 (I
believe CRM114 was called 'AWK on meth' or some such by someone in the
past?)

2- where to convert from byte/storage format to calculation/internal
wchar_t/int format? At the very start, so we don't have to convert
everywhere (less bug risk, probably less CPU load as well, when done
big bang style at the very beginning. But there's a tiny catch.[*])

3- what level of Unicode support do we plan for the next CRM114
decade? All the way, as in: (a) content plus (b) file names plus (c)
regexes or minus, say, (b)? (My take: all the way. Coding cost is
equal to (a) or (a)+(c) as a first guestimate.)

4- luck has it that the major use of CRM114 (my guestimate regarding
'major') is text classification, which, in regard to Unicode, has two
significant pieces of processing pipeline:

(a) tokenization through regex (that would be a Unicode match with
/preferrably/ a Unicode regex - TRE advertises it can do that) - no
extra performance cost apart from a few extra cache misses due to mem
footprint for data x 4 (see (1) above) and assuming we do NOT convert
the data every time we're feeding it to a regex matcher, but ONLY ONCE
at the start, i.e. when we hit 'window' or 'input' statements.

(b) feature hashing - which converts data irregardless of type to a
feature hash. Performance cost is that hashing unicode strings will
hash x 4 input /bytes/, which may be a minor/medium performance hit,
depending on classifier and hash algorithm used.


Summary
---------------

All in all, Unicode support should be doable with little extra effort
if we decide to convert to internal form of 1-int-per-char once and
make sure the code acts on that form throughout the run-time
(including TRE).

The guestimated costs are a worst-case RAM memory footprint of 4:1
compared to todays CRM114; since classifiers concern themselves with
feature hashes representing 'words' of one form or another, the file
storage space for CSS files should not be impacted for the production
classifiers. Some particular experimental classifiers may suffer a
file size increase of x 4 at worst case as well as they store the
literal content (FSCM, IIRC? Or was it Correlate?)

Run-time execution is guestimated to be a tad slower as well, mostly
due to the handling of the increased number of bytes (x4 @ worst case)
for content, but this should be minor when we stick each character in
a native integer (which, incidentally, would be 32-bit int at least) a
la gcc. No meager 16 bits for /our/ characters!

Porting effort for gcc-supported UNIXes would very probably be close
to zero then; Windows ports will require some effort and will very
probably only offer support for Unicode v3.0; which is okay for all
but the most academic uses, I suppose. (Okay, I never met a
Microsoft-based researcher of ancient Egypt contemplating the use of
CRM114 for his/her studies, so my field of vision is limited here.)


Full Unicode support also implies we tie ourselves rather strongly
with TRE and its further development, but then there hasn't been any
other regex library in the productive life of CRM114 anyway -- forget
that port to, what was it? PCRE? in there. The simple fact is that any
regex lib not capable of handling Unicode is not an option for the
future anyhow.



Which leaves the little catch:

[*] Since we are going to 'do' Unicode, are we also going to 'do'
encodings OTHER than UTF-8? (I can hear the stampede already...)
At bottom level, the basics are there: using libapr/libiconv et al
(which also come in Windows flavor, BTW)
At functional level though, I don't think you would love to see it
from a coding/support perspective: you can't just plonk it into the
crm114 input/window commands, and hopping back&forth using a
'alter'-like 'transform' command which must then recode faux Unicode
to bytes to codepage and back again is the best way I can see of doing
that without impacting anyone else's pipeline.
Would be handy for all those email/text-classifying folks out there, I
guess, but it's your call if you want to take it on or not. When you
think Unicode isn't bliss, I can assure you you'll loose all remaining
hair before the turn of the year when you include encodings as well.

Those who like smooching a lot ;-) may thus sensibly suggest to leave
the encoding/decoding of codepages to/from UTF-8 to external
applications exclusively.


Which sounds hunky dory, but then what about the image / byte
sequences processing folks (like me!), who do NOT use UTF-8 at all but
just raw bytes?
--> That means we have to support two 'encodings' at a minimum: UTF-8
and RAW. (RAW = expand each (unsigned) byte to wchar_t storage size on
input and vice versa on output)


My EUR 0.02 for today.

-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/