Re: match and UTF-8

On Sun, Jun 1, 2008 at 2:54 PM, Trever L. Adams <[email protected]> wrote:
> Am I understanding this correctly that match [utf-8 symbols] doesn't work? I
> am trying to do various currencies and I get some VERY bogus things with the
> following:
>
> match /[$£€¥元₨₪₱؋៛₡¢圓₩₭₮₨ƒ₦﷼฿₴₫£][[:space:]]*([[:digit:]]+)(\.[[:
> digit:]]*)?/
>
> This gives me responses in languages such as arabic, etc. even if I just use
> the first 4 and those symbols are no where to be found. All documents are in
> UTF-8 currently (and will be converted in the future when added/compared if
> they aren't).

Short answer:

UTF-8 is a 'problem' for CRM114. As are other encodings which include
characters which surpass the single byte (8 bits) when encoded.


Long (techno jock) answer:

Closer inspection of the source code regarding UTF-8 support leads to
these preliminary conclusions:

- CRM114 does not support 'code pages' or 'encodings' such as UTF-8
(though TRE can, IFF the right set of libraries is provided during
compilation and specific flags are passed at run-time when using the
non-wchar interfaces.)
- CRM114 is advertized as supporting full 8-bit; this is probably true
(I have not tested CRM114 extensively with full binary inputs) but
'full 8-bit range' is something else than supporting 'multibyte
encodings' such as UTF-8.

Bottom line: CRM114 known about single 'bytes', but does not really
know about the concept of 'character' is the human sense. It only
looks like it does when those 'characters' each fit into a single
8-bit byte.



So the 'hack' way is to convert all encoded 'characters' of interest
to 8-bit (single byte) characters for all inputs before feeding it to
CRM114 proper, OR upgrade CRM114 to full wchar ('wide character')
support[*].   <now see me make a beeline for the bomb shelter />



[*] TRE has that (wchar support) already, but that's only half the
story. Anyone who's ever spent time on 'upgrading' applications from
8-bit character to full wide character support will have some war
stories to share. CRM114 should be 'upgradeable', but that 'feature'
is not visible anywhere this side of the planning horizon, not for
GerH developments at least. So unless you sell Bill on it or  make it
a DIY... (Please note that wchar_t support means CRM114 suddenly will
have to 'understand' encodings such as UTF-8, UTF-16, etc. too; at
least one of them. So
 for f in *.[ch] ; do cat $f | sed -e 's/char/wchar_t/g' > new.$f ; done
is not going to cut it. Assume for example that you only 'tweak' the
interface to TRE to make it auto-convert assumed UTF-8 input; Q: who
guarantees your byte-offsets are actually on a _real_ 'character'
boundary? A: no-bo-dy.  --> the quick&dirty way is out for general
use.

From a technical viewpoint this is an 'interesting' challenge. And as
you probably know, any engineer uttering the word 'interesting' is
using an eufemism for 'this is going to blow your budget to kingdom
come' ;-)


So if you really want CRM114 to recognize those 'foreign' symbols, I
guess the sugeested' hack is the short-term solution: write a
preprocessor which converts the feed using a given encoding --> e.g.
produce UTF-32 (wide characters; single universal 'mapping' for
everything known out there), then pick all characters of interest to
you and convert them to a (custom?) single 8-bit byte 'encoding' and
encode all other (uninteresting) characters to a predefined 8-bit byte
value (e.g. '?' or something outside the ASCII range, e.g. 0xFF) so
that everything in your input is converted to
single-8-bit-byte-per-character. After which CRM114 will certainly be
able to handle that 'converted' feed.

Other solutions may be possible, but this way at least you can use
CRM114 as-is without loss of info - assuming the count of your
characters-of-interest plus one (for 'uninteresting character') are
less than 256 (using the NUL byte in your CRM114 data inputs *MAY*
lead to exhibiting yet uncovered bugs: it's kinda special in some
places).


Hope this helps, despite the delay in the response.


-- 
Met vriendelijke groeten / Best regards,

Ger Hobbelt

--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: [email protected]
mobile: +31-6-11 120 978
--------------------------------------------------
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general