Re: utf8u tag proposal
Osamu TAKEUCHI <[email protected]>
| Newsgroups | gmane.text.yaml.general |
|---|---|
| Message-ID | <[email protected]> |
Hi BlueG and William,
>>> This indeed declares a different type from !!str.
>> NO NO NO NO NO NO!!!! I will convert all strings TO "utf8u"
>> so they are IDENTICAL!!!
>
> It is a different type on the YAML end. Even if you convert them to and from
> the same native type in your application (or a library does so on your
> behalf) and/or don't use the !!str tag within the YAML presentation of that
> type.
>
> It is absolutely fundamental to the way YAML works that it defines its own
> data types. To be platform independent, it must, and does, define a common
> set of types (and also allows users to define their own types). What's more
> is, there is not always a one to one relationship between YAML types and
> native types.
>
> To demonstrate this last point, I'll show how I would map some basic types
> if I were writing my own YAML processor for C# (you'll need to view this
> with a fixed width font):
>
> +-----------+---------------+
> | C# Type | YAML Type/Tag |
> +-----------+---------------+
> | bool | !!bool |
> +-----------+---------------+
> | byte | !!int |
> | sbyte | |
> | short | |
> | ushort | |
> | int | |
> | uint | |
> | long | |
> | ulong | |
> +-----------+---------------+
> | float | !!float |
> | double | |
> | decimal | |
> +-----------+---------------+
> | string | !!str |
> | | !!utf8u |
> | | !!null |
> +-----------+---------------+
> | byte[] | !!binary |
> | | !!utf8u |
> | | !!null |
> +-----------+---------------+
>
> It doesn't take long to see that there are multiple problems (and I didn't
> even get into more complex types). How to map to and from native types is a
> problem that the YAML processor and application have to cooperate on. It is
> beyond the scope of the YAML specification or the definition of YAML tags.
> Oren has mentioned the idea of a schema language several times, and that is
> a way to go about it for more general purpose applications.
I do not agree with this table if the "YAML processor" is
for general purpose. I wish that a general purpose YAML library
is transparent through serialization and deserialization,
namely it preserves the data and the data type as far as
possible. Otherwise, it will not be widely used in real
applications, especially when we are talking about a library
for .NET.
See, my library always distinguish byte from ulong.
// Instantiates a serializer with the default configuration.
YamlSerializer serializer = new YamlSerializer();
// Put byte and ulong values into an untyped array.
object[] objs = new object[] { (byte)1, (ulong)1 };
// Serialize them.
string yaml = serializer.Serialize(objs);
// %YAML 1.2
// ---
// - !System.Byte 1
// - !System.UInt64 1
// ...
// Deserialize them.
object[] restored = (object[])( serializer.Deserialize(yaml)[0] );
// Confirm that the types are preserved.
Assert.IsFalse( restored[0].Equals(restored[1]) );
// ~~~~~~~ (byte)1 does not equal to (long)1.
For me, this is natural because when the users of the library
serialize a byte value into YAML, their main interest is not
which data type is of the YAML's standard, but is whether or not
their data is stored and restored without any modification.
Othewise, they will give an int value instead of a byte value.
Many-to-one type mapping by a library in order to use only
YAML's standard types seems to force the unwanted portability
to an application with unwanted pain to realize it. Umm, I have
to agree that I can not have one-to-one map to !!null for .NET,
as I have written the other day. I now think the one-to-many
mapping does not hurt the library users.
I give another example to make the issue clearer.
public class Test
{
public byte a = 3; // initial values are specified
public ulong b = 2.1;
}
string yaml = serializer.Serialize(new Test());
// %YAML 1.2
// ---
// !Test
// a: 3
// b: 2.1
// ...
Here, the Tags !System.Byte and !System.UIng64 are not specified
explicitly to the fields a and b but implicitly resolved from
the tag !Test to the parent node. They are not converted to !!int
for serialization.
So, regarding to the preservation of the original data and data type,
I understand William's desire. If I determine to strictly preserve
the content of string variables, for the users of the library,
I will have to serialize all the string variables as !!utf8u.
At this moment, I'm assuming the users do not require it, though.
> In any case, you might note that I list !!utf8u twice. There's a good reason
> for that. Firstly, I do it because a .NET string might contain invalid
> surrogate pairs that, if we want to preserve them without violating the YAML
> specification, would require storage using the utf8u type rather than the
> str type (though most applications would prefer the str type whenever
> possible). Secondly, on reading data in, the application may want the
> processor to translate utf8u scalars into strings whenever it can or it
> might even ask the library to translate invalid UTF-8 sequences in some way.
> Otherwise, the processor needs to use a byte array to store the original
> data as an unvalidated UTF-8 byte sequence. That means it also needs a way
> to communicate this fact to the application. Using Osamu's example class to
> encapsulate the byte array would be one way to go about this.
>
> My basic point, though, is that there isn't a one to one relationship
> between native types and YAML types. There is, in fact, a many to many
> relationship between native types. This isn't simply a matter of how the
> native type is encoded either. As stated, YAML needs to define a set of
> common types to be platform independent. Furthermore, it needs to define
> constraints on those types for them to be useful, which is why !!utf8u is
> different than !!str. !!str communicates a constraint that doesn't exist on
> !!utf8u. That constraint allows mapping !!str scalars to native string types
> in many cases where !!utf8u can not be (where !!utf8u must be treated as a
> byte array that needs decoded).
With reading this, I'm very much afraid that no general purpose
library will give a support to !!utf8u, because it is too
uncertain for me what is expected to a library. Thus, I do not
think it is a good idea to have !!utf8u in the tag repository
unless we have quite a lot of use cases where an independent
class similar to my example is beneficially used.
I do not deny that YAML can have its own data type. But when
no other language have such a data type, we have to think
carefully how it will be implimented in a library and used
in an real application. Note that it is almost hopeless to
build an application without a library, because of the
complexity of the YAML's syntax.
On the other hand, if we adopt the utf8u merely as a variation
of encoding, as the next,
- !!str @utf8u "..."
- !!binary @utf8u "..."
- !<!byte[]> @utf8u "..." # !< > is to have non ns-tag-char
# "[" and "]" in the tag
it will be more commonly implemented in libraries because
the role of a library is clear enough. I understand utf8u
should theoretically be independent of the real data type,
to which an utf8u data is stored (!!str, !!binary or !byte[]).
But I still think this will work much better in practice.
In this sense, I would like to have YAML's specification
practical than ideal. I wish YAML makes the real problems
easily solved than it is theoretically nicely defined.
Well, here, I assumed we had already had the encoding
property. Since I'm not sure if the encoding property is
really benefitial enough to modify the current YAML's
syntax, I have not reached the conclusion how the utf8u
should be defined in the YAML specification.
> As an aside, I'm still interested in the idea of encoding tags. Although
> !!utf8u might not be a valid candidate for an encoding tag (because the data
> it represents doesn't satisfy the constraints of the str type), I can see
> other cases where it might be useful, such as Osamu's earlier line feed
> problem. In general, however, I think some method for providing hints about
> the original data type would be more useful (was that !!null node an object,
> a string, a byte array, or what?). Schemas, whether defined implicitly or
> explicitly, can solve both of these issues, however. (I guess I've joined
> the schema bandwagon =p).
Yes, I was also thinking the same for the encoding property,
and reached almost the same conclusion at this point.
At first, I studied doing the next to solve the line break
preservation problem with !!utf8u,
string s = "abc\r\n def\r\nghi\r\n";
string yaml = serializer.Serialize(s);
// %YAML 1.2
// ---
// !!utf8u |2+
// abc%0d
// def%0d
// ghi%0d
instead of doing the next.
// %YAML 1.2
// ---
// "abc\r\n\
// \ def\r\n\
// ghi\r\n"
I saw the former is better but still less readable, and
it does not work if the line break is "\r". I would
like the next better if I had such an encoding property.
// %YAML 1.2
// ---
// @crlf |2+
// abc
// def
// ghi
But then, I would like to have a @crlf in front of evey
multi-line string, similarly as William wants to have his
string values all in !!utf8u. So, I thought of specifying
the default encoding for !!str as the next.
%ENCODING !!str @crlf
At this point, I realized this directive only theoretically
makes the document portable. I can find almost no use cases,
where a different application reads the document without the
directive and causes any problem because of the difference
in line feed. The only one case I found was that such an
application loads the file with line break normalization to
"\n" and saves the file in a double quoted text where the
line feeds are explicitly escaped as "abd\n def\nghi\n".
Except for this impractical anxiety, it is not too bad to
handle the document without the directive, with exchanging
the knowledge of the real linefeed format for the unescaped
multi-line strings in the YAML document, as the form of
implicit or explicit schema for the data.
So, I just added an option to my library to normalize all
unescaped line feed in the multi line text in YAML to any
form of line break. I now think this is the best way to
solve the problem, though the document will not be strictly
portable. Well, honestly speaking, I also had another point,
I didn't want to think of "!!str @crlf @utf8u". :(
For @utf8u, the same treatment causes the YAML document much
less portable, because we always have to unescape "%" to obtain
the original text. So, we will have to explicitly specify @utf8u
to each !!str node when the value is really escaped by "%".
It seems much better than specifying a !!utf8u in front of every
string value. Usually, there will be no @utf8u found in a YAML
document.
Regarding @binary, my library does not need !!binary nor @binary
even though it allows users to serialize an array of an arbitrary
value type in base64 encoding. The story is as follows.
My library encodes byte array as an sequence of byte values by
default, because it is a sequence of byte values. ;)
byte[] bytes = new byte[]{ 1, 2, 3, 4};
string yaml = serializer.Serialize(bytes);
// %YAML 1.2
// ---
// !<!System.Byte[]> [1, 2, 3, 4]
// ...
I looked for a nice definition of APIs with which we can
specify the base64 encoding for a specific object. But I
faild to it. So, my library allow users to use base64
encoding only when it is specified in the declaration of
class or structure, namely in the schema.
// .NET allows us to specify some custom metadata to
// the fields and properties of a class or structure.
public class TestClass
{
// base64 encoding is specified.
[YamlSerialize(YamlSerializeMethod.Binary)]
public byte[] LargeBinary = new byte[1024];
// base64 encoding is specified.
[YamlSerialize(YamlSerializeMethod.Binary)]
public float[] LargeFloatArray = new float[1024];
// base64 encoding is not specified.
public byte[] LargeBinaryWithoutBase64 = new byte[1024];
}
string yaml = serializer.Serialize(new TestClass());
// %YAML 1.2
// ---
// !TestClass
// LargeBinary:
// AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
// AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
// (snip)
// LargeFloatArray:
// AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
// AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
// (snip)
// LargeBinaryWithoutBase64: {0,0,0,0,0,0,0, (snip) ,0}
// ...
So, I currently have no need for @binary in my YAML library
because the parser always knows that the node is encoded
with base64 without an explicit node property.
Fmm, maybe there are very few use cases where the encoding
property should be explicitly specified.
However, we should realize that we have implicit encoding
properties anyway for these nodes. In addition, unless we have
somehow define the variety of encoding in the YAML specification,
library implementers can not determine whether or not it should
use base64 or any other format to encode some specific data.
Now, the definition of !!binary and other tags gives a regulation
(or just hints?) for the encoding format, even though their
original purpose was to define their own data type.
As another example, I would like to have @int encoding for
all native integer types of .NET and @float encoding for all
floating point types. I mean, accepting "10_000_000" as
!!int with rejecting "!System.UInt64 10_000_000" for ulong
(because .NET by default does not convert "10_000_000" into
ulong) will make the users confused. So I would like to
accept every YAML !!int style for every .NET integer type.
This can not be done unless !!int gives me a hint for the
standard encoding format for integer type values in YAML.
Umm, I myself might be a little confused about the
relationship between a YAML's data type and the accompanying
encording style. I'm also afraid to be going to mess up
the YAML specification, despite that I'm trying hard to
organize the relationship. :(
Best,
Osamu TAKEUCHI
------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf