Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)
Bill Yerazunis <[email protected]>
| Newsgroups | gmane.mail.spam.crm114 |
|---|---|
| Message-ID | <[email protected]> |
From: "Trever L. Adams" <[email protected]> Hello Bill, Yes, you are missing the point. It isn't just regex where UTF-8 would be good. The various "classifiers" all have their tokenization rules. I am not asking that they be made completely generic. The problem is that they assume various 8 bit sequences are ALWAYS a given mark. I would like to see a command line/library call that would enable UTF-8 tokenization. If everything is in UTF-8 (should be if they are using the call/command line), then it doesn't matter what the given text string means, it just matters where it is broken. OK, I am starting to understand. Would an example be: classify ( foo~funny_accent~bar.stat) [ my_text.txt ] where the ~funny_accent~ is some character that is (say) not in 7-bit ASCII? Everything else is (or should be) just numbers-and-spaces (that is, 0-9.-+E and space) or predefined flag values (i.e. defined in a .h file) If so, that's easy. There's only one place where things like that need to be dealt with and that's the filenames. Would this work: "in filename strings, the only delimiter characters are the classic whitespace characters (space, newline, carriage return, tab) and the succeed/fail separator vbar "|". Everything else is a filename." or am I still oversimplifying? If so, could you give me an example? -Bill ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/