Re: utf8u tag proposal

Hi William and BlueG,

>> So, I vote one to the form "!!utf8u a%b%c" as the canonical form.
> 
> I believe you meant "a%25b%25c". It was also suggested that the 
> canonical form replace *all* bytes with %xx, so it might be 
> "%61%25%62%25%63".

Excuse me for my carelessness.
Yes, I wanted to write "!!utf8u a%25b%25c".
I should not have written "canonical form" but 
I should have written "preferential encoding style".


>> Imagine my ruby application is using the YAML library...
> 
> I'm guessing Ruby is one of the many programs that uses UTF-16 or maybe 
> UCS-2 and calls it "Unicode". You are basically saying "the code must 
> not return something that will make my Ruby program throw an error". The 
> problem is that I think this is a bug in Ruby. Programmers and users 
> think of the conversion as a "cast" and do not expect errors. You are 
> attempting to patch this by changing every other api in the world that 
> can produce input data and artificially limit them to the subset that 
> won't throw errors. This is very bad programming practice and will never 
> work, for the simple reason that programs can have bugs, so assuming the 
> output is a given subset is impossible. The real solution is to fix 
> Ruby/Python so the "cast" really is just that. A second call to see if 
> the cast is lossy can be added.

At this moment, I do not understand your point very well.
Probably, I have to study the unicode encoding more.

But I'm also afraid that you misunderstood my post.

I did not intend to talk about any errors caused by a badly coded string
object nor how a badly (and nicely) coded unicode data should be stored 
in a native string object (UTF8 or UTF16 or error), though I had to show
some example to make the code meaningful. Instead, I wanted to discuss 
whether the library or an application should convert a !!utf8u node to 
a native string object. 

Usually, a !!str node is converted to a native string object seamlessly 
by a library. I thought the library should also do the conversion for 
!!utf8u. But you wrote the conversion by the library is not necessary. 
To clarify my point, I wrote that such a conversion must be done *before* 
applying the schema. So, when we are using a schema-oriented library, 
the conversion of a !!utf8u node to a native string object must be done 
either by the library itself or some callback function, instead of an 
application.

> William wrote at Fri, 25 Sep 2009 11:45:54 -0700:
>
>> Then, the library seamlessly converts the !!utf8u nodes back into 
>> string variables when deserialize.
> 
> This probably is not necessary. It is a lot easier for the application 
> to look for the tags and decode them. Also there seems to be an 
> absolute insistence that only valid utf-8 strings be returned by the 
> library which seems to make any such idea unacceptable to some here.



> I can tell you what I think a proper API for YAML is though I know it is 
> hopeless to convince people here:
> 
> 1. One call to return UTF-8. "UTF-8" means an array of bytes and 
> therefore this can return *any* array of bytes. This will return exactly 
> the byte stream in the file if the file is encoded in UTF-8, except for 
> the few ASCII characters that are part of yaml syntax.
> 
> 2. *Another* call to return UTF-16. The main purpose is to provide  data 
> that will not make Ruby/Python throw an error, though this may be 
> implemented more efficiently if the YAML file is UTF-16 encoded. If the 
> file is UTF-8 invalid bytes are decoded to 0xDCxx. The UTF-8 can still 
> be accessed with the other call, this is necessary as this call is lossy.
> 
> 3. A call to return "errors" with the current string. Although I think 
> you will be surprised at how little this will be used! Since yaml 
> scanned the string in order to parse the file, it can  often detect 
> these errors almost for free. One is invalid UTF-8, another is invalid 
> UTF-16. There can also be indicators for non-characters, control 
> characters, wrong canonical form, and all the other things that can be 
> "wrong" about a string.

If you really want to preserve both of badly coded UTF-8 and UTF-16,
I agree that your application needs two calls in the libaray API. 

But, I do not think it is required for a general purpose library. 
I expect a general purpose library to return the string in the system's 
default encoding. In ruby, we can set the system default encoding with 
$KCODE global variable. Since $KCODE can not be set "UTF16", I'm not 
sure if ruby really has its string objects in UTF-16. But, anyway, if 
ruby's unicode encoding is UTF16, the library should return UTF-16. 
If it is UTF8, it should return UTF-8.

I also did not see how you "call" the API. Are you going to have 
different versions of "load" functions?

obj = YAML.load_utf8( yaml )
obj = YAML.load_utf16( yaml )
obj = YAML.load_utf8u_error( yaml )


I am afraid that I'm misunderstanding your post...

Best,
Osamu TAKEUCHI

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf