Re: Unicode support (Was: Request for Comment: Callable CRM114 Classifiers (libcrm114))

On Thu, Sep 25, 2008 at 3:20 PM, Eugene Crosser <[email protected]> wrote:
> Makes perfect sense to me.
> So, for mail case, we would setup a pipeline like this:
>
> unmime (gets RFC822 message, MIME decodes it, and stuffs data into
> wchar_t buffer in accordance with "charset=XXX" attributes)
> ||
> \/
> tokenizer (gets wchar_t buffer, uses MB-capable RE library or other
> technique to select "features" and outputs them in the form suitable to
> the classifier)
> ||
> \/
> classifier[/learner] (gets sequence of features, produces result and/or
> updated state)
>
> For binary data, there will be no need for unmime, and tokenizer will be
> different (accepting void* binary data).
>
> (BTW, perfect case to use BIO approach ;-)
>
> Eugene


You're right on the money there. That's exactly what I was thinking.

This way, you can build any pipeline you want. email is a major one,
but you can thus do all sorts of wicked embedded app things as well:
pick packets from network interface chip or network protocol stack
layer, do a bit of minimal munching and spit it into classify.


(This does not yet answer how we're going to cope with the classify
output, but since that's at least going to be one format and
parseable, that 'hidden' issue wouldn't be too hard to cope with in
embedded or other non-usual environments. Besides, the 'error
feedback' hasn't been covered either, but I'll get to that in later
emails.)


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/