Re: utf8u tag proposal

BlueGM wrote:
> Its been a couple weeks now and no one has responded to this proposal...
> 
> I'd like to voice my support for it. I'd like to see this become a standard
> data type.
> 
> The basic definition (URI, Kind, Definition) all seem to be what we agreed
> on.
> 
> I'm not sure that as verbose a rationale is required (and I feel some of it
> asserts opinion as fact). Most of what needs to be said in the rationale is
> said in the first paragraph.

Probably correct to shorten to the first paragraph.

I still feel the biggest problem I am trying to solve is that users and 
programmers knee-jerk reaction to any encoding problem is to redefine 
the input as being ISO-8859-1 (or worse). This is causing a backsliding 
currently where systems are getting further from Unicode than they were 
a few years ago, something that is very worrying to me. Still trying to 
figure out a way to write this clearly and shortly.

Having just encountered ANOTHER piece of code that "fixed" encoding 
problems by stripping the high bit off every character just yesterday 
(this was to pass a string to Qt for display and was buried deeply in 
the code), I want to reiterate that this is a SERIOUS problem and 
pretending it does not exist is not helping!

> Explaining why the % sign is not always escaped
> is a bit misleading, however. Specifically, the problem I see is that in
> systems where "%%" is used to represent "%", the application would still
> need to be alert to cases where it was then followed by a pair of hex digits
> (as noted in the encoding notes), so saying that it provides compatibility
> with those systems is not true in all cases.

I did not explain this correctly. You could put "%25%25" in and it means 
two percent signs, this is not a problem. However as YAML is designed to 
be human-readable I wanted to allow the very commong "%%" escape used by 
higher level codes to be literal in the text, if "%%" is in the text 
then the data contains "%%", even though the user of that data may treat 
that as a single "%".

To disallow any temptation or erroneous implementations that treat "%%" 
as anything other than two percent signs, I purposly said that a '%' 
followed by any characters other than 2 hex digits is a literal '%'. The 
alternative of treating this as an error opens the possiblity for people 
to add "new" escapes to this encoding, I want them to be forced to use a 
new tag for that purpose.

> The encoding/decoding notes will be useful. For encoding point 2 ("Any
> sequence of bytes that is a valid UTF-8 encoding is written as the Unicode
> character (the UTF-8 encodings of U+D800..U+DFFF are *not* valid and must
> remain written as three %xx sequences)"), I think that this should say "any
> sequence of bytes that is the valid UTF-8 encoding of a printable
> character", which, per the YAML specification, excludes U+D800..U+DFFF as
> well as other invalid characters and those that don't have an easily
> discernable representation (those that can't be directly placed into a YAML
> file anyways).

I think valid encodings of characters that should not be in a YAML file 
should preferably be placed in as "\uXXXX" sequences. However this 
should not be done for the UTF-16 surrogate halves. The main reason is 
that YAML's normal strings do not like this, and also I believe it will 
assist in using this data form to store invalid UTF-16 as well.

> For the UTF-32 section, if we intend for it to be able to represent invalid
> UTF-32 (and I personally think this is of questionable value), then we
> should say that the UTF-8 representation is using the original specification
> for UTF-8 and not the more recent Unicode version (with its extra validation
> rules) somewhere in the definition. To my mind, this creates more problems
> than it solves since many languages already have encoders/decoders that work
> with the Unicode version of UTF-8, but might not work with the original
> version.

I think we can delete everything about UTF-32, except maybe a note in 
the UTF-16 section saying "similar approaches may be useful for storing 
invalid UTF-32".


------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf