Re: New invalid UTF-8 proposal using tags and %nn

On 9/3/09 2:21 PM, "William Spitzak" <[email protected]> wrote:

> I looked this up but it appears Perl is using "\x{NNNN}" as their
> version of what YAML is using "\uNNNN" for. It is not usable for
> producing an invalid encoding.

Yes it is. The Perl string "\x{FFFF}" or chr( 0xFFFF ) produce a single
character string with the code point 0xFFFF implemented as the octets
"\xef\xbf\xbf" which are UTF-8-like but invalid UTF-8.

Asking a proper encoder to render the octets outputs many warnings about the
"Malformed UTF-8 character" and eventually terminates in an "Unknown
encoding" exception.

Encode::encode( "\x{FFFF}", Encode::FB_CROAK )
Malformed UTF-8 character (character 0xffff) in ... at ... line ...
Unknown encoding '(the string)' at ... line ...

> The only "standards" I have seen for invalid UTF-8 is "\xNN" (from C,
> but incompatible with existing YAML), and "%nn" from URLs. I think I
> made a serious misstep in selling this idea originally by not following
> a standard and by proposing yet another escape sequence. I am now pretty
> convinced that some variation on %nn will work.

The "standard" as far as I'm aware of it is that perhaps you're allowed to
say invalid things if your environment lets you. I suspect YAML may go the
same route where your local YAML parsers and emitters may use your policy to
allow invalid but representable code points while other people may desire a
different policy that unrepresentable things are not safe for
interoperability and therefore invalid means invalid.

Josh


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july