Re: match and UTF-8

   From: "Ger Hobbelt" <[email protected]>

   On Sun, Jun 1, 2008 at 2:54 PM, Trever L. Adams
   <[email protected]> wrote:> Am I understanding this correctly
   that match [utf-8 symbols] doesn't work? I> am trying to do various
   currencies and I get some VERY bogus things with the> following:>>
   match /[$£€¥元₨₪₱؋៛₡¢圓
   ₩₭₮₨ƒ₦﷼฿₴₫£][[:space:]]*([[:digit:]]+)(\.[[:> digit:]]*)?/>> This
   gives me responses in languages such as arabic, etc. even if I just
   use> the first 4 and those symbols are no where to be found. All
   documents are in> UTF-8 currently (and will be converted in the
   future when added/compared if> they aren't).


   Short answer:
   UTF-8 is a 'problem' for CRM114. As are other encodings which includecharacters which surpass the single byte (8 bits) when encoded.

   Long (techno jock) answer: Closer inspection of the source code
   regarding UTF-8 support leads tothese preliminary conclusions:

   - CRM114 does not support 'code pages' or 'encodings' such as
   UTF-8(though TRE can, IFF the right set of libraries is provided
   duringcompilation and specific flags are passed at run-time when
   using thenon-wchar interfaces.)- CRM114 is advertized as supporting
   full 8-bit; this is probably true(I have not tested CRM114
   extensively with full binary inputs) but'full 8-bit range' is
   something else than supporting 'multibyteencodings' such as UTF-8.

   Bottom line: CRM114 known about single 'bytes', but does not
   reallyknow about the concept of 'character' is the human sense. It
   onlylooks like it does when those 'characters' each fit into a
   single8-bit byte.

This is correct.  CRM114 is blind about alignment of 8-bit bytes
in wchar or other frames.

There is always the question of "what encoding is the /RIGHT/ encoding
to view a particular byte sequence under".  The answer is, sadly, 
"It Depends...".

There are CRM114 applications that operate on byte sequences that may
contain bits that look exactly like UTF-8, or koi8-r, or JIS, or 
something else.  Is it "right" to shift over to other encodings when
it is possible to do so?  What about systems that can have
in-stream _unaligned_ embedded shifts, like Japanese?

Since some of the applications are _already_ in place that need to
_not_ auto-shift, the answer is "Nope.  We do not auto-shift".


   So the 'hack' way is to convert all encoded 'characters' of
   interestto 8-bit (single byte) characters for all inputs before
   feeding it toCRM114 proper, OR upgrade CRM114 to full wchar ('wide
   character')support[*].  <now see me make a beeline for the bomb
   shelter />

Yes.  The former is the recommended method.

You *can* embed anything you want into an isolated var by using the
\xHH escapes, then use that isolated var as a pattern in a match, and
all will be well EXCEPT that if the encoding you want to match against
is wchar-sensitive, you must somehow gaurd the regex such that it
doesn't match except when the alignment is as you have desired.

   [*] TRE has that (wchar support) already, but that's only half
   thestory. Anyone who's ever spent time on 'upgrading' applications
   from8-bit character to full wide character support will have some
   warstories to share. CRM114 should be 'upgradeable', but that
   'feature'is not visible anywhere this side of the planning horizon,
   not forGerH developments at least. So unless you sell Bill on it or
   make ita DIY... (Please note that wchar_t support means CRM114
   suddenly willhave to 'understand' encodings such as UTF-8, UTF-16,
   etc. too; atleast one of them. So for f in *.[ch] ; do cat $f | sed
   -e 's/char/wchar_t/g' > new.$f ; doneis not going to cut it. Assume
   for example that you only 'tweak' theinterface to TRE to make it
   auto-convert assumed UTF-8 input; Q: whoguarantees your
   byte-offsets are actually on a _real_ 'character'boundary? A:
   no-bo-dy.  --> the quick&dirty way is out for generaluse.  >From a
   technical viewpoint this is an 'interesting' challenge. And asyou
   probably know, any engineer uttering the word 'interesting' isusing
   an eufemism for 'this is going to blow your budget to kingdomcome'
   ;-)


   So if you really want CRM114 to recognize those 'foreign' symbols,
   Iguess the sugeested' hack is the short-term solution: write
   apreprocessor which converts the feed using a given encoding -->
   e.g.produce UTF-32 (wide characters; single universal 'mapping'
   foreverything known out there),    then pick all characters of
   interest toyou and convert them to a (custom?) single 8-bit byte
   'encoding' andencode all other (uninteresting) characters to a
   predefined 8-bit bytevalue (e.g. '?' or something outside the ASCII
   range, e.g. 0xFF) sothat everything in your input is converted
   tosingle-8-bit-byte-per-character. After which CRM114 will
   certainly beable to handle that 'converted' feed.  Other solutions
   may be possible, but this way at least you can useCRM114 as-is
   without loss of info - assuming the count of
   yourcharacters-of-interest plus one (for 'uninteresting character')
   areless than 256 (using the NUL byte in your CRM114 data inputs
   *MAY*lead to exhibiting yet uncovered bugs: it's kinda special in
   someplaces).

Why bother knocking it back to 8 bits?  If you leave it in wchar32's,
(and the first byte is always zero and none of the other bytes are)
then it will again be self-aligning and all will be well...

... until you try to use approximate regexes.  Then it's game over again.

    - Bill Yerazunis

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Crm114-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/crm114-general