Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

   The web pages are not all English. They range well into UTF-8 (and I 
   will be converting all to UTF-8 before processing them with CRM114). 
   This includes languages such as English, Romance (Spanish, French, 
   etc.), German, Japanese, Korean, Finnish, Swedish, Russian, etc. I would 
   prefer to have one training set, not one for each language. Right now 
   decimal 20 is always thought of as a space, even if the previous byte 
   had the hi-bit set (in UTF-8 means continues into next character, or 
   some such). Which would mean 20 was part of another character not a 
   space. The same may be true of just about any character. Right now the 
   regex won't match many UTF-8 monetary symbols due to the fact they look 
   like strings instead of one character. The classifiers will do 
   tokenization breaks in words because it sees a space or a punctuation 
   (or other legal token boundary) when it is really part of another word.

Ahhh!  This is possibly the problem.

If the Wikipedia article at:

   http://en.wikipedia.org/wiki/UTF-8

is correct, then that can never happen.  Basically we are just fine!

Assuming the web page is correct, UTF-8 works like this.  LATIN-1
characters are always 0-7F hex, and _no_ single-byte character is
allowed to have the high bit set.  Multibyte characters work with
a funky run-length encoding for up to four bytes (where x means
"this bit is available for information storage"):

 1-byte ASCII  (0x00-0x7F ) of the form 0xxxxxxx
 2-byte chars are of the form           110xxxxx 10xxxxxx
 3-byte chars are of the form           1110xxxx 10xxxxxx 10xxxxxx
 4-byte chars are of the form           11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

So, trailing bytes are _never_ allowed to contain a legal ASCII
character, because all legal ascii is forced to be 0-127 
only (never allowed to go into the 128-255 zone).  The trailing
bytes are always 10000000 or higher (128 and up) and legal ASCII
will never contain those bytes nor will any other legal UTF-8.

So, you will never see a false NULL or BLANK in the trailing bytes
of a UTF-8 stream; those characters simply are not legal 
codes to have in any byte of a multibyte UTF-8 stream.

    [...]
   I am seeing this happen a lot, especially with NULLs (0x00) and 0x20 
   (space) which are not really there, but are UTF-8 sequences.

If the Wikipedia article is to be believed then you are seeing defective
UTF-8.  NULL and SPACE should _never_ be in the multibyte stream.

Are you sure you are not seeing UTF-16 or CESU-8 instead?  The Wikipedia 
article says that that is a common glitch.

I personally haven't dealt with UTF-8 much, so I'm going on what the spec
says, not what I know by debugging. 

   So, say crm114 had an option like: crm -utf8. That should change the 
   calls to the regex engine to do utf-8 processing. It should also change 
   the tokenizer in leaners and classifiers to do UTF-8 safe parsing 
   (again, it only needs to know what is really the real token boundaries 
   and not part of another character...) and break the words right and 
   process the .crm file correctly so I can say, include monetary symbols 
   in UTF-8 in the program and have it understood correctly.

   UTF-8 seems to be the way most of the world is going. It is common on 
   most foreign language (non-english) websites in place of most other 
   encodings (at least in my experience). It is the base encoding used for 
   ODT (OpenOffice) and many other things. It seems to be the common coding 
   now for emails which use other than LATIN-1, especially if going to 
   someone who may speak multiple languages, etc.

   I think the technical world has spoken, UTF-8 is the future. And, for 
   LATIN-1/ASCII it is the exact same thing (0-127 anyway). Yes, for those 
   who aren't using it, they may not want the overhead, hence a library 
   call and/or a command line switch (AND preferred as you seem to be 
   making a library interface for this all now).

   It has been a while since I viewed the UTF-8 standard, but it seems to 
   me the tokenizing would be easy. Simply look at the previous byte, if it 
   had the hi-bit set (>127) then this byte is part of that character 
   sequence, add it to the string we are building. If it wasn't and we have 
   a valid token boundary, break and move onto the next token. As for the 
   regex, sorry, I don't know TRE at all.

Not quite.  

>From what I see, it looks like everything works fine with ONE exception;
if you hand any regex engine a byte stream that contains UTF-8, then
you get regexes that match that UTF-8.  Or at least you should; if it
doesn't then the regex engine is buggy.

The one exception is that if you have a numeric count capability then
you can't just use indexing; you have to actually examine the byte
stream to see how many characters it contains (basically, bytes 0x00
to 0x7F are one character, bytes 0xC0 to 0xFF are one character, and
bytes from 0x80 to 0xBF are zero characters - but must be included 
with their leading 0xC0 to 0xFF bytes.

As far as I can remember, there is only one such place in CRM114 that
does NOT go thorugh the regex engine, and that's in input variable
restriction.  Everything else is either inside a regex, or is
part of POSIX file I/O which actually does count bytes, not "characters",
or is the "string length" operator which counts bytes, not characters.

But clearly you've been bitten by this bug more than once.  Can you
give an example?

     - Bill Yerazunis

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/