Re: utf8u tag proposal

Hi all,

Since I started this, I would like to explain why I withdrew 
the proposal, a little more. I think this discussion is also 
good to clarify what is expected to a YAML library to implement 
!!utf8u and !!binary. At this moment, my main concern about 
utf8u is that very few general purpose libraris will provide
support for it because it is uncertain what should be done by 
library and how. I hope this discussion helps for it.

>> Another concern is about the node's equality.
>> I would like to have the next two nodes equal to each other.
>> - !!utf8a "a"
>> - "a"
>>
>> But, in the current proposal, they will not be equal.
> 
> I agree with Osamu that this is a huge problem with the current proposal 
> and design of YAML.

I understood that the !!utf8u type is supporsed to be something 
like the next.

class utf8u
{
  public byte[] data;
  bool is_valid_utf8();
  string convert_to_utf8_if_possible();
}

Since utf8u data can not be safely stored in a utf8 string variable,
the raw data is stored in byte array. This class is only used 
temporarily after the data is input from user and before it is
validated to be a valid utf8 string for the future use.


This indeed declares a different type from !!str. Since YAML is
not aware of the comparison between different types, we do not 
have to be afraid about the comparison between !!utf8u and !!str.
This is an analogy of comparing !!int 0 and !!float 0.


> If I want to obey strict YAML rules, I would have to put !!utf8u in 
> front of EVERY string, whether it had valid UTF-8 in there or not. I 
> only want to use the tag if in fact there is invalid UTF-8 in the 
> string, but assuming some other program does not agree with what is 
> "valid" then it may produce unequal data for no good reason.

We have to put !!utf8u tags only in front of evely utf8u object.
When our data is already stored in a string variable, there is no 
need to treat it as utf8u in YAML. The reason is, the main purpose 
of utf8u is not to pass a badly encoded utf8 data to some other 
APIs that possibly causes some serious problem with such an irregal 
input.

In addition, if your application does not need a freehand portability,
instead of doing,

- !!utf8u "..."
- !!utf8u "..."
- !!utf8u "..."
- !!utf8u "..."

you can also think of doing

!<!utf8u[]>
- "..."
- "..."
- "..."
- "..."

here the !!utf8u tags are resolved by the application specific
!utf8u[] tag.


> I suspect "!!binary" has worse problems in that it is even more likely 
> to have different types, and users are forced to embed the type into the 
> binary data, where other yaml processors cannot see or use it.

I think YAML provided !!binary just as the representation of byte 
array and it is not supporsed to be used to encode some other type
of data in BASE64 format. Namely, we should use !!binary only when 
we encode byte array.

Yes, I also expected some more for !!binary at first, though.


> In any case you have to realize that both of these tags are wanted as 
> another method of quoting strings. Just like whether a string is flow or 
> quoted does not enter into it's equality, this should not either.

So, what I realized is that it is not allowed to use !!binary unless
the original data is byte[] and to use !!utf8u unless the original
data is some special class like the utf8u class I showed above.


> Therefore some ability to make multiple tags, some of which only control 
> the encoding, and others which are both encoding and type, are needed.

I do also think that it is useful to have some means to specify binary 
encoding to other types. But I also understand the current specification 
is selfconsistent in the sence I discussed above. This is why I withdrew
my proposal, though I myself would like to have "Node Encoding" property.


> My proposal was "!!taga !!tagb ...", and I guess I implied that if there 
> are more than 1 the last one (at least) is an encoding-only tag.
> 
>> It can be an option to treat the next two to be equal,
>> though it is incompatible to the current definition.
>>
>> - !!str#utf8a "a"
>> - "a"
> 
> This is interesting except if you want to be compatible '#' cannot be 
> used. Tags can only have these punctuation marks:
> 
>  '_', '-', ';', '/', '?', ':', '@', '&', '=', '+', '$', ',', '.', '!', 
> '~', '*', '\'', '(', ')', '[', ']'

I did not understand this.

 [99] c-ns-shorthand-tag ::= c-tag-handle ns-tag-char+

 [40] ns-tag-char        ::= ns-uri-char - “!” - c-flow-indicator

 [39] ns-uri-char        ::= “%” ns-hex-digit ns-hex-digit | ns-word-char | “#”
                    | “;” | “/” | “?” | “:” | “@” | “&” | “=” | “+” | “$” | “,”
                    | “_” | “.” | “!” | “~” | “*” | “'” | “(” | “)” | “[” | “]” 

 [23] c-flow-indicator   ::= “,”  | “[”  | “]”  | “{”  | “}”

ns-tag-char seems to contain "#".


> You could select two, one for normal tags and one for encoding-only 
> ones. Maybe '!' for normal and '?' for encoding-only, and replace the 
> leading '!' with it as well, so in normal use the proposed tag would be 
> written as "!?utf8u".
> 
>> Then, instead of having data type and encoding both in the
>> Tag property, how about defining another Node property,
>> "Node Encoding" or something.
>>
>> - !!str @utf8a "a"
> 
> Since this is incompatible I don't see any improvement over reusing '!' 
> for this.

Note that !?utf8u or anything is anyway incompatible with an
existing parser. In addition, there might be many applications that 
are already using ? or # in their tags. So, if we determin ? or # 
to be a new special character in a tag, the new specification will
be incompatible to the possibly existing such applications.

On the other hand, I understand @ is a reserved indicator. 
We can use it when the majority of this comunity want to have
additional syntax component in YAML.


I'm still thinking about the use cases for !!binary and !!utf8u or 
@binary and @utf8u, and the way a library gives support for them.
But it will take some time for me to overview the issues.

Best,
Osamu TAKEUCHI

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf