Re: utf8u tag proposal

Osamu TAKEUCHI wrote:

> Hence, we should not serialize our data
> 
>  [ "a%b%c", "d%e%f" + badcode, "g%g%i" ]
> as the next.
> 
>  - !!utf8u a%%b%%c
>  - !!utf8u d%%e%%f%XX%XX   # %XX%XX denotes the bad code
>  - !!utf8u g%%h%%i

I want to make clear that I ABSOLUTELY DO NOT WANT "%%" to be the escape 
for '%'. The escape is "%25". I want the definition to say that '%' 
followed by anything other than 2 hex digits is a literal '%', this is 
on purpose so nobody is tempted to say "%%" is an escape! So these 
examples are written like this:

  - !!utf8u a%25b%25c
  - !!utf8u d%25e%25f%XX%XX
  - !!utf8u g%25h%25i

The output converter can also check if literal percent works and output 
this, this is my recommendation:

  - !!utf8u a%b%c
  - !!utf8u d%e%f%XX%XX
  - !!utf8u g%h%i

(an example where literal percent does not work is "%abc" because there 
are 2 hex digits, this would have to be written as "%25abc"

The reason for this is so that a YAML program that ignores the !!utf8u 
tag will get as close as possible to the original string, and so it is 
fairly easy using a text editor to remove/replace the bad codes and the 
tag and turn the bad string into a good one.

> At least, I and probably William expect that a library does the first 
> use case automatically, namely it surveys all the string values and 
> stores bad ones in !!utf8u node without asking the application.

Yes I think this would be far easier to use than the application having 
to look at it, mostly because application writers screw up. Also a lot 
more efficient because libyaml already scans the strings. And it can 
correctly implement the decision whether to escape % signs, and 
correctly recognize whether codes are legal or not. And the library 
could intelligently decide whether to quote or not using different rules 
than when writing normal strings, because some characters that normally 
require quoting can be put in as %XX (this is rarely preferred but there 
were examples where where it was used to put \r at the end of each line).

It does mean however that if there is a tag set by the program it will 
have to produce an error or be replaced, this is why we are requesting 
the idea of multiple tags. Though really I am not using tags on any 
string data (just maps) and if this is normal it may not be a big problem.

> Then, 
> the library seamlessly converts the !!utf8u nodes back into string 
> variables when deserialize.

This probably is not necessary. It is a lot easier for the application 
to look for the tags and decode them. Also there seems to be an absolute 
insistence that only valid utf-8 strings be returned by the library 
which seems to make any such idea unacceptable to some here.

It would however be a million times easier to use this way!

Bill Spitzak
Rhythm & Hues Software

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf