Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

On Mon, Sep 22, 2008 at 11:36:08AM -0600, Trever L. Adams wrote:
> >
> > Do you mean in the regex engine?
> >
> > If so, then it's not something libcrm114 itself handles.  We accept
> > whatever the regex engine itself accepts (and that includes locality

TRE handles wchars:

...
Wide character and multibyte character set support

   TRE supports multibyte character sets. This makes it possible to
   use regexps seamlessly with, for example, Japanese locales. TRE
   also provides a wide character API.

Binary pattern and data support

   TRE provides APIs which allow binary zero characters both in
   regexps and searched strings. The standard API cannot be easily
...

problem is, that _we_ don't (yet). So you (afaikt) can't set a lexer as
a wchar regex.

> Yes, you are missing the point. It isn't just regex where UTF-8 would be 
> good. The various "classifiers" all have their tokenization rules. I am 

no, the rule is one: a regex lexer (frontend tokenizer) that chops input 
8-bit text. Thereafter the code is (should be!) token-agnostic.
  
> not asking that they be made completely generic. The problem is that 
> they assume various 8 bit sequences are ALWAYS a given mark. I would 

problem is, that 1byte = 1char, which is not true on wchar stream.

> Obviously, what I say only applies to those designed for text. The 
> various binary learners would not or should not be affected.

they are, as long as you expect the lexer is something wchar-meaningful,
instead of a raw 8bit sequence - eg you want to chop a mmedia stream at
some wchar text markers/tags.


-- 
paolo

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/