Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

On Mon, Sep 22, 2008 at 01:20:30PM -0600, Trever L. Adams wrote:

> prefer to have one training set, not one for each language. Right now 
> decimal 20 is always thought of as a space, even if the previous byte 

nope, seems _you_ have missed something here :) - it's up to you to define
your lexer, [[:graph:]+] is just the default when you don't set it
explicitly via the /<lexer regex>/ syntax.
I'm not sure it'd always work for wchar text, but the simplest actions
like

{
  learn (...) /<lexer regex>/
}

{
  classify (...) /<lexer regex>/
}

where /<lexer regex>/ is 'coded' as explict byte sequences should work.
For that to work with an UTF-8 editor used to code the .crm, (at least)
the parser and compiler need to deal properly with wchar (I think) and 
that's _not_ the case (I think).
But if you can 'code' your regex via \x.. sequences as Bill pointed out
that should work. 

And if you have UTF-8 strings and an UTF-8 editor you can try and see
what happens.

-- 
paolo

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/