Re: FW: Re: Invalid UTF-8

On Thu, 2009-09-03 at 17:19 -0700, Josh ben Jore wrote:
>     Do you copy?  As of Perl 5.8.7, B<UTF-8> means strict, official UTF-8
>     while B<utf8> means liberal, lax, version thereof.  And Encode version
>     2.10 or later thus groks the difference between C<UTF-8> and C"utf8".
> 
>       encode("utf8",  "\x{FFFF_FFFF}", 1); # okay
>       encode("UTF-8", "\x{FFFF_FFFF}", 1); # croaks

Just like "only *p*erl can parse *P*erl" and so on. Sigh.

This is "true, but mostly irrelevant". The proposed tag is not
compatible with Perl's "universal" utf8 encoding, because Perl's utf8
encoding does not process %nn escape sequences.

This is why I think it is best to call this tag !!utf-u. The 'u' could
be taken to be either 'Universal' or 'Url-encoded' or both. This is
exactly what it is: a (universal) UTF encoding combined with URL
encoding.

As for %DEFAULT...SCALAR directives (suggested by BlueGM): Adding a new
standard tag is easy and has no effect on the spec. We just add it to
the tag repository and we are done. Adding a directive, on the other
hand, is a *huge* deal, and brings us to YAML 1.3 territory. So this is
not on the table for a "long while" at this point.

In addition I think this directive a serious overkill. It is a blunt
instrument and I can see many problems with it. A much better approach
would be to work on a schema language for YAML that would allow one to
specify how tags are associated with nodes at a much more controlled and
fine grained manner.

As far as '%20' goes: I strongly believe that '%20' should be a space
and not '%' '2' '0' inside such !!utf-u. The only argument against it
was mangling of URLs, and it fails on two accounts.

First, there's simply no reason to ever use this tag for encoding URLs.
All the URLs (actually all the URIs) in the world can be easily
processed as normal YAML strings. They have their own (%nn) built-in
escape mechanism. { path: "file://foo%ff" } _already_ works, without
having to annotate it with !!utf-u. So why would you ever want to?

Second, even if you decided to pass a URL inside a !!utf-u tag (for some
strange reason), the fact that %20 would be preserved but %80 would be
mangled to %2580 is _extremely_ confusing. Requiring % to always be
escaped as %25 is a no-brainer; it is simple, consistent and follows the
rule of least surprise.

Have fun,

    Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july