UTF-8

Bill Yerazunis wrote:
> If the Wikipedia article is to be believed then you are seeing defective
> UTF-8.  NULL and SPACE should _never_ be in the multibyte stream.
>
> Are you sure you are not seeing UTF-16 or CESU-8 instead?  The Wikipedia 
> article says that that is a common glitch.
>   
My mistake. As I said, it has been a while and I read a bunch. Glad to 
know that the text learning/classification is working fine.

> Not quite.  
>
> From what I see, it looks like everything works fine with ONE exception;
> if you hand any regex engine a byte stream that contains UTF-8, then
> you get regexes that match that UTF-8.  Or at least you should; if it
> doesn't then the regex engine is buggy.
>
> The one exception is that if you have a numeric count capability then
> you can't just use indexing; you have to actually examine the byte
> stream to see how many characters it contains (basically, bytes 0x00
> to 0x7F are one character, bytes 0xC0 to 0xFF are one character, and
> bytes from 0x80 to 0xBF are zero characters - but must be included 
> with their leading 0xC0 to 0xFF bytes.
>
> As far as I can remember, there is only one such place in CRM114 that
> does NOT go thorugh the regex engine, and that's in input variable
> restriction.  Everything else is either inside a regex, or is
> part of POSIX file I/O which actually does count bytes, not "characters",
> or is the "string length" operator which counts bytes, not characters.
>
> But clearly you've been bitten by this bug more than once.  Can you
> give an example?
>
>      - Bill Yerazunis
>   

Alright, examples:


# Normalize currency
:normalize_currency:
{
isolate (:mycount: :mycount2: :tocount: :tocount2:)
    {
        alter (:tocount2:) //
        alter (:tocount:) //  
        alter (:mycount:) //
        # crm114 doesn't seem to understand utf-8
        # currency symbols: £€¥元₨₪₱؋៛₡¢圓₩₭₮₨ƒ₦﷼฿₴₫£ # These used to be 
with the $ below
        match <fromnext> 
/[$][[:space:]]*([[:digit:]]+)(\.[[:digit:]]*)?/ (:keep: :tocount: 
:tocount2:)
#        output /ORIGKEEP: :*:keep: TO COUNT: :*:tocount: :*:tocount2: 
COUNTS :*:mycount: :*:mycount2:\n/
        eval (:mycount:) /:@: :#:tocount: :/
        eval (:mycount2:) /:@: :#:tocount2: -1 :/
        alter (:keep:) /$/
        {
            eval (:mycount:) /:@: :*:mycount: - 1 :/
            eval / :@: :*:mycount: >= 0 :/
            alter (:keep:) /:*:keep:X/
            liaf
   
        }
        {
            eval / :@: :*:mycount2: >= 0 :/
            alter (:keep:) /:*:keep:./
        }
        {
            eval (:mycount2:) /:@: :*:mycount2: - 1 :/
            eval / :@: :*:mycount2: >= 0 :/
            alter (:keep:) /:*:keep:X/
            liaf   
        }
#        output /KEEP: :*:keep: TO COUNT: :*:mycount: :*:mycount2:\n/
        liaf
    }
}
return

The above doesn't work right. I believe it matches any given byte and 
not the utf-8 currency symbols themselves. If I was doing a string 
instead of a "give me any character found in" it would probably work. 
For the problem areas search for currency symbols above and read it and 
the next line.

Sorry for all the trouble and misunderstanding of UTF-8 and CRM114.

Trever

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEUEARECAAYFAkjdKbUACgkQTmpiivh5HXM8uwCfQtB3nmZswnyMzLPSG4n1iHFP
TPkAmNmEXUzfoHSivjbP6uQXp+TF2vM=
=O6i/
-----END PGP SIGNATURE-----