Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

On Tue, Sep 23, 2008 at 08:31:33AM -0400, Bill Yerazunis wrote:

> is correct, then that can never happen.  Basically we are just fine!

nope, see also Ger's/mine posts - basically we're cheating, treating 
UTF*/wchar as binary data, thus ignoring/bypassing regex-engine (TRE)
wchar API, eg you can match a specific binary string, but you've no
classes, upper/lower -case and so on.

> If the Wikipedia article is to be believed then you are seeing defective
> UTF-8.  NULL and SPACE should _never_ be in the multibyte stream.

broken text/stream for whatever reason may happen in real world.

> Are you sure you are not seeing UTF-16 or CESU-8 instead?  The Wikipedia 
> article says that that is a common glitch.

that's the point, dealing with 'just binary' means doing by hand what wchar
libraries/API are supposed to do already. That's dirty and cheap if you
have a limited set of strings (say something like a subset of HTML tags)
but likely won't work in general.

> The one exception is that if you have a numeric count capability then
> you can't just use indexing; you have to actually examine the byte
> stream to see how many characters it contains (basically, bytes 0x00
...
> part of POSIX file I/O which actually does count bytes, not "characters",
> or is the "string length" operator which counts bytes, not characters.

this may have deep implications - while the wchar API should make the regex
engine return properly wchar-bounded pointers, this might not happen in
binary mode. Also, while I'm pretty fine with latin1 only scripting (is 
C UTF* ready? PERL? awk? others?) I'm concerned wrt crm/scripting in UTF*
environment: the arg parser uses \x20, strtol() for offsets and so on -
what happens editing/running in UTF-8 environment? ever screwed up badly
doing your normal stuff in uterm (UTF-8 xterm)? I did. Because that '5' I
was reading wasn't just that but a multibyte thing that neither me nor
the app/script I was playing with knew how to handle.


-- 
.p.a.o.l.o

[that's UTF8 in latin1 xterm ;) ]


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/