Re: utf8u tag proposal

Hi All,

I'm not strongly denying this proposal but have some
concern on it. Namely, I think we have to be aware of 
the two groups of Tags in the Tag repository.

The first group consists of all the existing tags in the
repository except for !!binary, !!null and !!utf8u. They
specify the data type of a node. I wondered if I should 
classify !!merge, !!value and !!yaml to this group but 
I at last understood they define special classes whose 
instance is used only when we need some placeholder for 
the key of a special key-value-pair in a mapping.

The second group consists of !!binary and !!utf8u, and
possibly !!null. These Tags do not specify the data type 
but the way of encoding the native data into its string 
expression.


So, sometimes, we want to specify both the tags in the 
first and the second group, scimultaneously to one
data node. This is the reason why William proposed 
allowing set of tags to one node in his post on 2009-09-04.

> I'm thinking that allowing a *set* of tags might work, 
> though I am not sure if this will parse. For instance:
> 
> - !!MyClass "ascii representation"
> - !!MyClass !!binary "adf89asdf00asdfasdf..."

I do not support "set of tags" because it will be very 
much incompatible to the existing parsers. Instead, 
I was considering to have the tags(?) in the second
group as a part of a tag.

- !!string#utf8u "..."
- !SomeClass#utf8u "..."
- !gif#binary "..."
- !<!byte[]#binary> "..."    # !< > is to allow the non ns-tag-char's
- !<!float[]#binary> "..."   # "[" and "]" to appear in the tag.

Note that we might be able to encode !gif data also
into YAML like the following.

- !gif
  - Header: GIF89a
  - Logical Screen Descriptor:
    - width pixels: 3
    - height pixels: 5
    - ...

http://en.wikipedia.org/wiki/Graphics_Interchange_Format#Example_.gif_file



It is not obvious but !!null is similar to !!binary for most 
languages with strict type-binding (except for the one like 
ruby that have a special class for null value), because !!null 
nodes have their own data types, though it might be void*.

The next example assumes we have a C# class BinaryTree as
follows.

  public class BinaryTree
  {
    public string Value;
    public BinaryTree ChildA;
    public BinaryTree ChildB;
  }

Note that classes in C# is very different from those in C++.
They are more like the objects in ruby. An instance of C# 
classes is always a reference of objects. So, we can have 
null value in the fields ChildA and ChildB.

Then, the next YAML can be naturally interpreted.

- !BinaryTree
  Value: "ghi"
  ChildA: 
    Value: "def"
    ChildA: 
      Value: "abc"
      ChildA: null  # This !!null is in reality !BinaryTree#null.
      ChildB:       # This is also !BinaryTree#null.
    ChildB: ~       # This too.
  ChildB:
    Value: "jkl"
    ChildA: NULL    #
    ChildB: Null    #

In this case, !!null merely defines the way how we should 
encode null value of *some reference type* into string. 
In this sense, I write !!null is similar to !!binary, though, 
for ruby, it really represent the singleton class Nil and 
its only one instance.


Another concern is about the node's equality.
I would like to have the next two nodes equal to each other. 

- !!utf8a "a"
- "a"

But, in the current proposal, they will not be equal. 
It can be an option to treat the next two to be equal,
though it is incompatible to the current definition.

- !!str#utf8a "a"
- "a"


Then, instead of having data type and encoding both in the
Tag property, how about defining another Node property,
"Node Encoding" or something.

- !!str @utf8a "a"
- "a"
- !!string @utf8u "..."
- !SomeClass @utf8u "..."
- !gif @binary "..."
- !<!byte[]> @binary "..."
- !<!float[]> @binary "..."
- !SomeClass @null null
- !SomeClass null      # Note that @null is a defalut 
                       # encoding for reference types. 
                       # So, we don not have to specify 
                       # it always.

Fmm, this looks completely fine for me, now.


The above proposal is not my conclusion. I haven't thought 
of enough number of use cases yet. Anyway, I do not think 
it is a good idea to invent many tags in the second group 
before we determine our way regarding to the issues: 
how we specify the data type and the encoding style to a 
node scimultaneously and how we define the equality of 
same data in different encodings.

Best,
Osamu TAKEUCHI



> Its been a couple weeks now and no one has responded to this proposal...
> 
> I'd like to voice my support for it. I'd like to see this become a standard
> data type.
> 
> The basic definition (URI, Kind, Definition) all seem to be what we agreed
> on.
> 
> I'm not sure that as verbose a rationale is required (and I feel some of it
> asserts opinion as fact). Most of what needs to be said in the rationale is
> said in the first paragraph. Explaining why the % sign is not always escaped
> is a bit misleading, however. Specifically, the problem I see is that in
> systems where "%%" is used to represent "%", the application would still
> need to be alert to cases where it was then followed by a pair of hex digits
> (as noted in the encoding notes), so saying that it provides compatibility
> with those systems is not true in all cases.
> 
> The encoding/decoding notes will be useful. For encoding point 2 ("Any
> sequence of bytes that is a valid UTF-8 encoding is written as the Unicode
> character (the UTF-8 encodings of U+D800..U+DFFF are *not* valid and must
> remain written as three %xx sequences)"), I think that this should say "any
> sequence of bytes that is the valid UTF-8 encoding of a printable
> character", which, per the YAML specification, excludes U+D800..U+DFFF as
> well as other invalid characters and those that don't have an easily
> discernable representation (those that can't be directly placed into a YAML
> file anyways).
> 
> For the UTF-32 section, if we intend for it to be able to represent invalid
> UTF-32 (and I personally think this is of questionable value), then we
> should say that the UTF-8 representation is using the original specification
> for UTF-8 and not the more recent Unicode version (with its extra validation
> rules) somewhere in the definition. To my mind, this creates more problems
> than it solves since many languages already have encoders/decoders that work
> with the Unicode version of UTF-8, but might not work with the original
> version.
> 
> Thanks,
> 
> BlueG
> 
> 
>  
> 
>> -----Original Message-----
>> From: William Spitzak [mailto:[email protected]] 
>> Sent: Monday, September 07, 2009 10:33 PM
>> Cc: [email protected]
>> Subject: [Yaml-core] utf8u tag proposal
>>
>> Okay I tried to type up a formal proposal for the !!utf8u 
>> tag. I based this on a !!binary proposal I found.
>>
>> Sorry about the plain text but I figured this would make it 
>> easier for somebody to put it in a wiki.
>>
>> Comments/changes please!
>>
>>
> 
> 
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay 
> ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
> http://p.sf.net/sfu/devconf
> _______________________________________________
> Yaml-core mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/yaml-core


------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf