Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

On Thu, Sep 25, 2008 at 11:53 AM, Eugene Crosser <[email protected]> wrote:
> In fact, I meant "all that is above libcrm114" which may be a pipe of
> some re-encoder and the crm114 binary. I have no problem if the crm114
> binary only support one specific encoding (provided that it covers all
> languages). Although *if* external encoding (like UTF-8) is *not* going
> to be the same as internal (like, MBCS aka wchar_t aka mb*() RTL
> functions), I see no difference in implementing one specific encoding
> vs. any encoding supported by system locale.
>
> There is another story if libcrm114 will use some "external" encoding
> (like UTF-8) *internally*. I.e. if libcrm114 will *not* accept wchar_t*
> data, but rather char*, and require that this char* buffer be a UTF-8
> stream. I do not mean to object it, it's up to you people who understand
> how the data is used inside the classifiers, to decide. But still, I'd
> like to point that most likely, any RE library (as well as libc
> infrastructure: wcslen(), wcscmp() etc.) will require that the data is
> passed in MBCS/wchar_t/"internal" representation. Because this is how
> the unicode infrastructure is designed to work: always use wchar_t
> inside C program, and only convert it from/to any suitable "external"
> encoding (including but not limited to UTF-8) on input/output. So, if
> you want to use MBCS support of the RE library, you'll need to convert
> data to MBCS anyway.
>
> Whether MBCS support in the RE library is needed at all is a separate
> question. It *is* needed if we want expressions like '/[aàá]bcd/'.
> Do we want them or not - I do not know.

Yup. It's a tough call. Your reply has made me think again. Thanks. ;-)

Things get a little hazy here because crm114 has two important
sections when it comes to data processing:

- the classifiers
- the regex-assisted text processing facilities (in the crm114 script language).

For the latter, it generally makes sense to assume we are processing
'text' and that word means we've got to cope with code pages one way
or another.

The former (the classifiers) can also be said to work on 'text', but
really they don't: they work on 'features', which take various forms:
hashes, bits, ...

Let's ditch the text processing/scripting section for a while when
thinking about it and hence assume libcrm114 will only offer
classifiers. (That'ld mean we might have a libcrmscript or simply the
crm114 app itself if we want those text manipulating facilities too)

Given that the classifiers don't need 'text' input to work -- they
just need 'data' and a way to chop that 'data' into 'atoms' (called
'features'; BitEntropy then uses 1-bit 'features') -- your reply has
given me an idea that feels rather better than going for either UTF8
or wchar_t: libcrm114 should not assume any 'encoding' AT ALL.

In technical terms, that would mean libcrm114 classifier APIs should
exhibit a mem*() like interface behaviour instead of a
str*()/mbcstr*()/wstr*() one: void* pointers for data.
Plus callbacks, which can be configured by the libcrm114 user (=
application developer) as [s]he will know what kind of data and
possible encoding(s) will be fed to the classifiers: the only function
block in any classifier that is concerned with the *format* of the
data is the tokenizer (be it either a regex-assisted word extractor
feeding the VT engine or a bit-chopper or ...): hence the way out is
to have the tokenizer exist as an external (user defined) part, fed to
the libcrm114 classifers with the data the tokenizer has to work on.

As Bill pointed out, the hash algorithm may be adversely affected by
wchar_t input. No problem there: the hash function is also part of the
tokenizer, as classifiers are not interested in 'words' or
'characters' (those are only 'intermediate data') as classifiers want
to receive 'features'.

A *classifier* doesn't care if the features it receives were conceived
from RAW, MBCS, UTF-8 or wchar_t originals or whatnot; we munch
'features' only, be it 32-/64-bit hashes or single bits.


For ease of use of libcrm114 some 'often used tokenizers/feature
hashers' can be provided with it, but that's an extra.

Thanks for your remarks! They really helped clear the fog in my head.
(I still wasn't happy with the Unicode/encoding stuff either way.)


This way, libcrm114 does not have to worry about encodings, language
support or Unicode: those bits are implementation specific and should
be provided from outside the library.
(Take my own data analysis case, where I work on sampled data, which
are numbers (signal levels really). *I* don't need nor want Unicode
for my crm114 use, because it only adds a conversion layer that I will
have to 'undo': twice the work for zero effect. The fact that most
other folks will be munching email just means everyone else will just
use a different 'tokenizer' than me; while all of us use the same
libcrm114.)


Lemme see if I can whip up a little crude sample, so we (and Bill
especially) can see this work...






-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/