Re: Request for Comment: Callable CRM114 Classifiers (libcrm114)

Bill Yerazunis wrote:
> OK, I am starting to understand.
>
> Would an example be:
>
>   classify ( foo~funny_accent~bar.stat) [ my_text.txt ]
>
> where the ~funny_accent~ is some character that is (say) not
> in 7-bit ASCII?    Everything else is (or should be) just
> numbers-and-spaces (that is, 0-9.-+E and space) or predefined
> flag values (i.e. defined in a .h file)
>
> If so, that's easy.  There's only one place where things like that
> need to be dealt with and that's the filenames.
>
> Would this work: 
>
>   "in filename strings, the only delimiter characters are the
>    classic whitespace characters (space, newline, carriage return, tab)
>    and the succeed/fail separator vbar "|".  Everything else is a filename."
>
> or am I still oversimplifying?
>
> If so, could you give me an example?
>
>    -Bill
>   
Sorry, Bill. You are still oversimplifying.

The example: as you may recall, I am working on a web page 
classification system. There were 60 categories, I have shrunk that 
somewhat.

The web pages are not all English. They range well into UTF-8 (and I 
will be converting all to UTF-8 before processing them with CRM114). 
This includes languages such as English, Romance (Spanish, French, 
etc.), German, Japanese, Korean, Finnish, Swedish, Russian, etc. I would 
prefer to have one training set, not one for each language. Right now 
decimal 20 is always thought of as a space, even if the previous byte 
had the hi-bit set (in UTF-8 means continues into next character, or 
some such). Which would mean 20 was part of another character not a 
space. The same may be true of just about any character. Right now the 
regex won't match many UTF-8 monetary symbols due to the fact they look 
like strings instead of one character. The classifiers will do 
tokenization breaks in words because it sees a space or a punctuation 
(or other legal token boundary) when it is really part of another word.

The only languages I speak use LATIN-1, (single byte UTF-8) so I cannot 
give you an example. But lets say the following string required at least 
two bytes and _ means a space which is not a space, but part of a 
character encoding.

"My_shoes ate my_dog yesterday"

Right now, CRM114 sees "my", "shoes", "ate", "my", "dog", "yesterday". 
But this is wrong, UTF-8 would show "my_shoes" (whatever the actual 
characteres would be, again, bogus example), "ate", "my_dog", "yesterday".

I am seeing this happen a lot, especially with NULLs (0x00) and 0x20 
(space) which are not really there, but are UTF-8 sequences.

So, say crm114 had an option like: crm -utf8. That should change the 
calls to the regex engine to do utf-8 processing. It should also change 
the tokenizer in leaners and classifiers to do UTF-8 safe parsing 
(again, it only needs to know what is really the real token boundaries 
and not part of another character...) and break the words right and 
process the .crm file correctly so I can say, include monetary symbols 
in UTF-8 in the program and have it understood correctly.

UTF-8 seems to be the way most of the world is going. It is common on 
most foreign language (non-english) websites in place of most other 
encodings (at least in my experience). It is the base encoding used for 
ODT (OpenOffice) and many other things. It seems to be the common coding 
now for emails which use other than LATIN-1, especially if going to 
someone who may speak multiple languages, etc.

I think the technical world has spoken, UTF-8 is the future. And, for 
LATIN-1/ASCII it is the exact same thing (0-127 anyway). Yes, for those 
who aren't using it, they may not want the overhead, hence a library 
call and/or a command line switch (AND preferred as you seem to be 
making a library interface for this all now).

It has been a while since I viewed the UTF-8 standard, but it seems to 
me the tokenizing would be easy. Simply look at the previous byte, if it 
had the hi-bit set (>127) then this byte is part of that character 
sequence, add it to the string we are building. If it wasn't and we have 
a valid token boundary, break and move onto the next token. As for the 
regex, sorry, I don't know TRE at all.

Thanks,
Trever

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/