Re: utf8u tag proposal

Hi BlueG,

Thanks for your intense response.

I'll respond only about utf8u in this post.
Other issues will be done in an independent posts 
because they seem getting independent of utf8u.

> In any case, utf8u does not fit in this category. It must have its own type
> because it does not obey the constraints on the type defined by the !!str

I think this sentence made me catch the point correctly.
I now understood that !!utf8u can not be merely an encoding for 
a !!str node, because !!str node can never contain bad coding 
anyway. At the same time, we should distinguish this fact from 
the issue whether or not a native string variable can contain
badly coded utf8. It completely makes sense.


Then, let me study how !!utf8u works in reality. 

Imagine we are worring about having some badly coded utf8 in our 
set of string data in our string variables of some language. 
Probably the situation is some thing like the following. We 
expect user input in utf8 and almost always that is true. 
But we can not say almost always is really always. So, if 
we pass such a data to a YAML library which is not concious of 
such bad coding, the resulting YAML might not be a valid YAML 
document. In order to avoid it, we should use !!utf8u node to 
store such a value. 

However, if we store all our string values in !!utf8u nodes, our 
YAML document will be no longer portable. Yes, we can say it is 
theoretically portable because !!utf8u is defined in the YAML 
specification. But, in the reality, since few real applications 
understand !!utf8u, we will suffer from doing so. 

Hence, we should not serialize our data

  [ "a%b%c", "d%e%f" + badcode, "g%g%i" ] 

as the next.

  - !!utf8u a%%b%%c
  - !!utf8u d%%e%%f%XX%XX   # %XX%XX denotes the bad code
  - !!utf8u g%%h%%i

Instead, we should store our string value in a !!utf8u 
node only when it really contains some bad coding. 

  - a%b%c
  - !!utf8u d%%e%%f%XX%XX
  - g%h%i

Now, our document is almost always compatible to other 
applications, except for the cases where it really contains 
any badly coded values in some node. We can accept this 
incompatibility because such a document have been anyway 
incompatible to YAML.

To do this, we probably have to implement the code by 
ourselves. When serializing, our code surveys all the string 
variable and stores some of them into !!utf8u nodes while storing 
others in !!str nodes. When deserializing, it decodes all !!utf8u 
values back into string variables so that we preserve the content
strictly. On the library side, !!utf8u is treated as same as
a user defined local tag. Users convert their string values to 
!!utf8u node on their own responsibility and pass the node to 
the library.

The other use case has been always discussed where another user 
might have possibly badly coded utf8 data in a binary block rather 
than in a string variable. So, he converts all such binary blocks 
into !!utf8u when serializing. When deserialize, !!utf8u tags are
converted back to the binary blocks. A library again do not know 
what is done for !!utf8u outside of the library.

If we imagine these two use cases, we can not see how a library 
can give a support for !!utf8u. 

Now, I doubt the second use case is really common.


At least, I and probably William expect that a library does the 
first use case automatically, namely it surveys all the string 
values and stores bad ones in !!utf8u node without asking the 
application. Then, the library seamlessly converts the !!utf8u 
nodes back into string variables when deserialize. Automatically 
creating !!utf8u node when serializing is not bad because, unless 
doing so, the library can not output a valid YAML document anyway. 
When deserializing, it have to avoid surprizing an application by 
the badly coded string when the application does not expect for it. 
So, the library should ask the application if it should convert 
!!utf8u to string value probably with some call back function. 

If this is the major use case for !!utf8u, I expect not a few 
libraries will give supports for it. It is very clear what should 
be done. It is useful for almost all users and seems not harmful 
for any application. If this can be a major use case of !!utf8u, 
we should promote how a library is expected to make use of !!utf8u.


Note that, in order to solve the second use case, he or she can 
still do the same with denying conversion from !!utf8u to native 
string, if the library has such an APIs.

Best,
Osamu TAKEUCHI


------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf