Re: utf8u tag proposal

> Probably correct to shorten to the first paragraph.
> 
> I still feel the biggest problem I am trying to solve is that 
> users and programmers knee-jerk reaction to any encoding 
> problem is to redefine the input as being ISO-8859-1 (or 
> worse). This is causing a backsliding currently where systems 
> are getting further from Unicode than they were a few years 
> ago, something that is very worrying to me. Still trying to 
> figure out a way to write this clearly and shortly.
> 
> Having just encountered ANOTHER piece of code that "fixed" 
> encoding problems by stripping the high bit off every 
> character just yesterday (this was to pass a string to Qt for 
> display and was buried deeply in the code), I want to 
> reiterate that this is a SERIOUS problem and pretending it 
> does not exist is not helping!


I know how you feel about that, but, if someone was going to implement this
data type, it's a lot of extra wording that doesn't change the task at hand.


> I did not explain this correctly. You could put "%25%25" in 
> and it means two percent signs, this is not a problem. 
> However as YAML is designed to be human-readable I wanted to 
> allow the very commong "%%" escape used by higher level codes 
> to be literal in the text, if "%%" is in the text then the 
> data contains "%%", even though the user of that data may 
> treat that as a single "%".


My point was simply that, the way it is worded, someone might read it and
think that a "%%" will always be interpreted as "%%", but this won't be the
case if it is followed by two hexadecimal digits. For example, "%%cat". If
someone thought that "%%" was always "%%", then they might think this was
going to be "%%cat" when, in fact, it is "\xCAat" (in double-quoted syntax).
I understood, and agree with, your reasoning for allowing a "%" that isn't
followed by two hexadecimal digits to be interpreted simply as "%". We just
need to be careful about the way it is worded in the rationale.


> I think we can delete everything about UTF-32, except maybe a 
> note in the UTF-16 section saying "similar approaches may be 
> useful for storing invalid UTF-32".


I would agree. Because it is dependent upon how the application decodes the
UTF-8 (whether or not the library it uses, if any, supports the full 32 bits
of the original UTF-8 specification), it isn't very cross-platform. More
over, there just aren't many use cases for invalid UTF-32 that can't be
translated into a UTF-8 or UTF-16 problem.


At this point, most of my concerns about the draft are word choices and the
verbosity of the rationales. The core audience for the data type is
engineers who need to implement or use the type, so their primary interest
is in how to do that and they often, rightly or wrongly, assign precise
meanings to words (which makes sense since it is their job to extract a
precise meaning from the specification).

I've been hoping to hear what others say before going into depth about it
though (which will probably require creating a version with mark up).


------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf