Re: utf8u tag proposal

Osamu TAKEUCHI wrote:
> Hi William and BlueG,
> 
>>> So, I vote one to the form "!!utf8u a%b%c" as the canonical form.
>>
>> I believe you meant "a%25b%25c". It was also suggested that the 
>> canonical form replace *all* bytes with %xx, so it might be 
>> "%61%25%62%25%63".
> 
> Excuse me for my carelessness.
> Yes, I wanted to write "!!utf8u a%25b%25c".
> I should not have written "canonical form" but I should have written 
> "preferential encoding style".
> 
> 
>>> Imagine my ruby application is using the YAML library...
>>
>> I'm guessing Ruby is one of the many programs that uses UTF-16 or 
>> maybe UCS-2 and calls it "Unicode". You are basically saying "the code 
>> must not return something that will make my Ruby program throw an 
>> error". The problem is that I think this is a bug in Ruby. Programmers 
>> and users think of the conversion as a "cast" and do not expect 
>> errors. You are attempting to patch this by changing every other api 
>> in the world that can produce input data and artificially limit them 
>> to the subset that won't throw errors. This is very bad programming 
>> practice and will never work, for the simple reason that programs can 
>> have bugs, so assuming the output is a given subset is impossible. The 
>> real solution is to fix Ruby/Python so the "cast" really is just that. 
>> A second call to see if the cast is lossy can be added.
> 
> At this moment, I do not understand your point very well.
> Probably, I have to study the unicode encoding more.
> 
> But I'm also afraid that you misunderstood my post.
> 
> I did not intend to talk about any errors caused by a badly coded string
> object nor how a badly (and nicely) coded unicode data should be stored 
> in a native string object (UTF8 or UTF16 or error), though I had to show
> some example to make the code meaningful. Instead, I wanted to discuss 
> whether the library or an application should convert a !!utf8u node to a 
> native string object.
> Usually, a !!str node is converted to a native string object seamlessly 
> by a library. I thought the library should also do the conversion for 
> !!utf8u. But you wrote the conversion by the library is not necessary. 
> To clarify my point, I wrote that such a conversion must be done 
> *before* applying the schema. So, when we are using a schema-oriented 
> library, the conversion of a !!utf8u node to a native string object must 
> be done either by the library itself or some callback function, instead 
> of an application.
> 
>> William wrote at Fri, 25 Sep 2009 11:45:54 -0700:
>>
>>> Then, the library seamlessly converts the !!utf8u nodes back into 
>>> string variables when deserialize.
>>
>> This probably is not necessary. It is a lot easier for the application 
>> to look for the tags and decode them. Also there seems to be an 
>> absolute insistence that only valid utf-8 strings be returned by the 
>> library which seems to make any such idea unacceptable to some here.
> 
> 
> 
>> I can tell you what I think a proper API for YAML is though I know it 
>> is hopeless to convince people here:
>>
>> 1. One call to return UTF-8. "UTF-8" means an array of bytes and 
>> therefore this can return *any* array of bytes. This will return 
>> exactly the byte stream in the file if the file is encoded in UTF-8, 
>> except for the few ASCII characters that are part of yaml syntax.
>>
>> 2. *Another* call to return UTF-16. The main purpose is to provide  
>> data that will not make Ruby/Python throw an error, though this may be 
>> implemented more efficiently if the YAML file is UTF-16 encoded. If 
>> the file is UTF-8 invalid bytes are decoded to 0xDCxx. The UTF-8 can 
>> still be accessed with the other call, this is necessary as this call 
>> is lossy.
>>
>> 3. A call to return "errors" with the current string. Although I think 
>> you will be surprised at how little this will be used! Since yaml 
>> scanned the string in order to parse the file, it can  often detect 
>> these errors almost for free. One is invalid UTF-8, another is invalid 
>> UTF-16. There can also be indicators for non-characters, control 
>> characters, wrong canonical form, and all the other things that can be 
>> "wrong" about a string.
> 
> If you really want to preserve both of badly coded UTF-8 and UTF-16,
> I agree that your application needs two calls in the libaray API.
> But, I do not think it is required for a general purpose library. I 
> expect a general purpose library to return the string in the system's 
> default encoding. In ruby, we can set the system default encoding with 
> $KCODE global variable. Since $KCODE can not be set "UTF16", I'm not 
> sure if ruby really has its string objects in UTF-16. But, anyway, if 
> ruby's unicode encoding is UTF16, the library should return UTF-16. If 
> it is UTF8, it should return UTF-8.
> 
> I also did not see how you "call" the API. Are you going to have 
> different versions of "load" functions?
> 
> obj = YAML.load_utf8( yaml )
> obj = YAML.load_utf16( yaml )
> obj = YAML.load_utf8u_error( yaml )

More like:

obj = YAML.load(...)
bytearray = obj.asUTF8()
wordarray = obj.asUTF16()
bool = obj.hadUTF8Error()
bool = obj.hadUTF16error()

This may be a poor understanding as I am using the libyaml event api, 
not the structural one. The obj is the event.

> I am afraid that I'm misunderstanding your post...

It's possible we are both asking for the same thing: we want to convert 
to native strings without the possibility of producing an error. 
Currently this requires an api to be provided that returns a native 
string, because of erroneous implementations of languages such as Python 
so that it is difficult otherwise to get an error-free conversion.

I am suggesting that instead of saying "native string api" that might 
return UTF-8 or UTF-16, there is instead explicitly "UTF-16" and "UTF-8" 
apis. This is a LOT more useful for programmers, as most/all languages 
support arrays of 16-bit and 8-bit objects and thus can call the 
non-native api as well as the one they need. And pretending that 
programmers don't know or need to know the native representation is 
stupid and arrogant and absolutely false.

Now ideally the implementers of Python and Ruby, etc, need to wake up 
and realize how people want to handle text. We DO NOT WANT ERRORS!!! 
Similar to how converting double to int does not throw errors for 
numbers out of range or NaN, because implementors quickly realized that 
this is incredibly impractical and useless, despite the theoretical 
correctness. Instead there is isnan() and other calls, which you *can* 
use to detect the possible errors. Or simply test int(x)!=x to see if
there was loss during the conversion. We want a "convert UTF-8 to native 
string" that NEVER throws an error, and a DIFFERENT call that is "that 
conversion was lossy".

Fortunatly UTF-8 is a superset of UTF-16 (all UTF-16 including invalid 
ones can be losslessly converted to UTF-8), which means that cross-type 
equality comparisons can be done by converting to UTF-8. Therefore 
unicode(x)==x will always be false if the conversion was lossy.

But until programming languages are fixed, we must provide the extra API 
to return "something that won't throw an error". You called this "native 
string", I am calling it "UTF-16".

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf