Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

Ger Hobbelt wrote:

> In technical terms, that would mean libcrm114 classifier APIs should
> exhibit a mem*() like interface behaviour instead of a
> str*()/mbcstr*()/wstr*() one: void* pointers for data.
> Plus callbacks, which can be configured by the libcrm114 user (=
> application developer) as [s]he will know what kind of data and
> possible encoding(s) will be fed to the classifiers: the only function
> block in any classifier that is concerned with the *format* of the
> data is the tokenizer (be it either a regex-assisted word extractor
> feeding the VT engine or a bit-chopper or ...): hence the way out is
> to have the tokenizer exist as an external (user defined) part, fed to
> the libcrm114 classifers with the data the tokenizer has to work on.

Makes perfect sense to me.
So, for mail case, we would setup a pipeline like this:

unmime (gets RFC822 message, MIME decodes it, and stuffs data into
wchar_t buffer in accordance with "charset=XXX" attributes)
||
\/
tokenizer (gets wchar_t buffer, uses MB-capable RE library or other
technique to select "features" and outputs them in the form suitable to
the classifier)
||
\/
classifier[/learner] (gets sequence of features, produces result and/or
updated state)

For binary data, there will be no need for unmime, and tokenizer will be
different (accepting void* binary data).

(BTW, perfect case to use BIO approach ;-)

Eugene

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI25BBfrw/cIw6UWkRAoxxAJ4oxUjX6j19hbfS53kdmqccM5ZivwCg05ll
KF/1UIry86hwRuLkASVfRQM=
=G/R9
-----END PGP SIGNATURE-----