Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

   From: Paolo <[email protected]>

   On Tue, Sep 23, 2008 at 08:31:33AM -0400, Bill Yerazunis wrote:

   > is correct, then that can never happen.  Basically we are just fine!

   nope, see also Ger's/mine posts - basically we're cheating, treating 
   UTF*/wchar as binary data, thus ignoring/bypassing regex-engine (TRE)
   wchar API, eg you can match a specific binary string, but you've no
   classes, upper/lower -case and so on.

I don't understand how you can get spaces or nulls inside the UTF-8
multibyte characters and have it be "correct" UTF-8.  None of the LATIN-1
characters have the high bit set, and high-bit-set is required 
for multibyte characters in UTF-8.


   > If the Wikipedia article is to be believed then you are seeing defective
   > UTF-8.  NULL and SPACE should _never_ be in the multibyte stream.

   broken text/stream for whatever reason may happen in real world.

Yeah, but if it's already broken, there's not much hope for it.

Consider: if we see 0xE0 in a UTF-8 stream, then we are in a
three-byte multibyte character, and the first four bits of that
multibyte are 0000.  The next two bytes in the stream _must_ be of the
form 10xxxxxx (with six data bits each); hence the only valid values
in the stream are 0x80 through 0xBF for the next two bytes.  _NOTHING
ELSE IS VALID_ for the next two bytes.

And - attempting to reassemble a Unicode codepoint from the broken
values low order six bits is madness.  I.e. you see a 0x79 "y") then
what do you do?  Take the low-order six bits?  That's 0x39 "9".
That's not right.  

So we take the alternative of just masking off the high order bit; now
0x79 remains "y". 

But what do we do with the four bits 0000 we already have from the
prior 0xE0 byte?  Throw them away?  Bit-extend them into a NULL?  
Data is gone here; we cannot recover correctly.

Now we see some _correct_ UTF-8 which happens to have a next byte
of 0xB0.  Do we take this byte as correct (i.e. six bits, value 100000) or
do we say "nope, that's really an ASCII space?".

As I see it, we can't win.  UTF-8 was specifically designed to "resynch"
at the start of each real character, even on garbles.  But when we're in
a garble, it's really not easy (or necessarily possible) to recover the
data.  

I think the best we can do is use the UTF-8 resynchronization; there
was a character there but it makes no sense; we just move forward with
charcount incremented.


   > Are you sure you are not seeing UTF-16 or CESU-8 instead?  The Wikipedia 
   > article says that that is a common glitch.

   that's the point, dealing with 'just binary' means doing by hand what wchar
   libraries/API are supposed to do already. That's dirty and cheap if you
   have a limited set of strings (say something like a subset of HTML tags)
   but likely won't work in general.

   > The one exception is that if you have a numeric count capability then
   > you can't just use indexing; you have to actually examine the byte
   > stream to see how many characters it contains (basically, bytes 0x00
   ...
   > part of POSIX file I/O which actually does count bytes, not "characters",
   > or is the "string length" operator which counts bytes, not characters.

   this may have deep implications - while the wchar API should make the regex
   engine return properly wchar-bounded pointers, this might not happen in
   binary mode. 

Ahhh... wchar is emphatically NOT the same as UTF-8.

Rather, wchar_t is a fixed width; it's 16-bits on windows and 32-bits 
on GCC.  It does NOT do variable-length multibyte like UTF-8 does.


   Also, while I'm pretty fine with latin1 only scripting (is 
   C UTF* ready? PERL? awk? others?) 

no language that I'm aware of allows anything except LATIN-1 as
variable names (including APL!).

   I'm concerned wrt crm/scripting in UTF*
   environment: the arg parser uses \x20, strtol() for offsets and so on -
   what happens editing/running in UTF-8 environment? ever screwed up badly
   doing your normal stuff in uterm (UTF-8 xterm)? I did. Because that '5' I
   was reading wasn't just that but a multibyte thing that neither me nor
   the app/script I was playing with knew how to handle.


   -- 
   .p.a.o.l.o

   [that's UTF8 in latin1 xterm ;) ]

No, it can't be... or more accurately, if it had multibyte characters,
it did not get through email successfully.  It shows as 0x2E 0x70 0x2E
0x61 0x2E 0x6F 0x2E 0x6C 0x2E 0x6F (total 10 bytes) and none of the
bytes have the high order bit set.  In short, it's plain LATIN-1
one-byte characters (which is a proper subset of UTF-8, because none
of the high-order bits are set indicating a multibyte character).

	 - Bill Γerazunis 

(Yes, that _should_ be a capital "Gamma" at the start of my last name;
wc does show that my last name is 10 bytes but the terminal shows only
nine character spaces used up.  If this xterm in UTF-8 mode is working
correctly and the mail system does the right thing, everyone should see
a cap-Gamma, and wc on a cut/paste of my last name should show 10 bytes
used.)




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general