Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)
Bill Yerazunis <[email protected]>
| Newsgroups | gmane.mail.spam.crm114 |
|---|---|
| Message-ID | <[email protected]> |
From: Paolo <[email protected]> > not asking that they be made completely generic. The problem is that > they assume various 8 bit sequences are ALWAYS a given mark. I would problem is, that 1byte = 1char, which is not true on wchar stream. > Obviously, what I say only applies to those designed for text. The > various binary learners would not or should not be affected. they are, as long as you expect the lexer is something wchar-meaningful, instead of a raw 8bit sequence - eg you want to chop a mmedia stream at some wchar text markers/tags. Ahhh... OK. _That_ I can understand. Hmmm... But- if the Wikipedia article on UTF-8 is to be believed, then I still don't see a problem. The number of bytes (that is, the counted-length) is just fine. The character set is also just fine (spaces are still 0x20, vertical bars are still 0x7C, and so on). And - all UTF-8 that isn't ASCII is gauranteed to have all of the first bits starting 11xxxxxx and the following bytes being 10xxxxxx, so again, regexes will (or should) work fine as long as they just pretend the UTF-8 is really just 8-bit stuff. How about an example to help me see this? - Bill Yerazunis ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/