Re: Invalid UTF-8

On Thu, 2009-08-20 at 18:44 -0700, Adrian Klaver wrote:
> It is a data issue, pushing it up the stack only prolongs the agony.

Exactly. Not only prolongs it but also spreads it everywhere. That said,
in cases like (say) UNIX files names containing arbitrary bytes, there's
not much you can do "data entry"-wise.

BTW - Unicode does allow for _all_ 8-bit characters. You can argue about
their semantics, but the fact is that a simple "\xNN" escape sequence
inside double-quoted strings _will work_ for all the 256 single-byte
values. So, for the case of stuff like UNIX file names, I really don't
see the problem. As for tags - I don't see anyone defining tags that are
not valid printable Unicode characters. It isn't as if we need to
support all the imaginable UR*L*s out there. Tags are a very specific
controlled set of UR*I*s.

You can even use this to encode arbitrary binary data, as long as you
accept that (1) you are using a 4 YAML stream bytes ('\' 'x' N N) for
each 8 bits of payload (except for say 1/4 of the bytes - call it 3
bytes on average for random binary data), (2) that there will be an
interim in-memory string representation using one "character worth" of
bytes for each 8-bit payload (2 bytes if using UTF-16 like Java does,
1-2 bytes if using UTF-8 - call it 1.5 bytes on average for random
binary data), and (3) that you'll need to execute string2binary on the
interim in-memory string to convert it to a true binary bytes array.

You'd be better off using !!binary and base64 for true binary data, of
course - only 1.25 YAML stream bytes for each 8 bits of payload, and it
will be loaded directly to an in-memory bytes buffer with no interim
representation. But still, using "\xNN" for binary data may be a valid
trick to use in some circumstances.

Either way, I simply don't see the use case for supporting invalid
Unicode characters, either as raw bytes or as escaped characters.

Have fun,

    Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july