Re: Next YAML: drop equality definition

Zenaan,

Thank you for your comment.
It let me know that I should have shown how I use custom
tags and custom tag resolutions in my apps.

Indeed, the C# library I developed uses the class
definitions as the custom schema.

Namely, if a YAML document

   !MyClass { a: true, b: true }

is given to the library and MyClass is defined as

   class MyClass
   {
     public boolean a { get; set; }
     public string b { get; set; }
   }

the library read the first true as boolean, the second
true as string and construct a MyClass object from the
document, correctly. It refers to the class definitions
by using C#'s reflection ability.

So, even a document

   !<!MyClass[]>
   - { a: true,  b: true }
   - { a: false, b: truf }
   - { a: true,  b: trug }

can be processed correctly without needing !MyClass
tags explicitly on all the items of the sequence
because the library can see !MyClass[] is an array
of !MyClass. (Surrounding !< > is just needed to
escape special characters of '[' and ']' in the
tag name)

Moreover, if you let the library know what is the
expected tag for the root node, the only tag in
the document can be also omitted as:

  - { a: true, b: true }
  - { a: false, b: truf }
  - { a: true, b: trug }

This document can be read safely and correctly as
an array of MyClass with the given _schema_. In this
sense, my YAML library is fully _schema-aware_.


This type of tag resolution is allowed in the
current YAML spec. With this library, users can
handle YAML document with their own custom schema
very easily.

In particular, it works nicely to serialize/
deserialize a complex object graph that consists of
variety of native/custom classes into a YAML document.
YAML's tag system perfectly fits for the purpose.

I admit the document is not readable for
schema-blind people or tools. They will not see that
the properties a and b are of different type. But
it is not evil. When one wants to read or edit any
YAML document, they must be familiar with the document.
They must know the purpose of the document and types
of all nodes referring to the exact schema with which
the document is written.

This is how the schema works in my apps.

Do you think my library is exceptional? Such a library
can be build with any programing languages with the
reflection ability. If such libraries are provided
for Ruby, javascript, Scala and etc., YAML will be
more popular as serializing language.

With such use cases in my mind, I give comments below.


On 2016/03/10 22:01, Zenaan Harkness wrote:
> On 3/10/16, Osamu TAKEUCHI <[email protected]> wrote:
>>> 2. YAML data is portable between programming languages.
>>>
>>> Different applications can apply a different schema to
>>> the same document.If these schemas differ on assigning
>>> identity then you cause all sort ofsticky issues.
>>
>> What do you mean by "can"? Could you give us any example
>> where different applications _can_ apply a different
>> schema to the same document?
>
> Such as application/ layer 4 "application" with application specific
> schema, and schema-blind middleware, and in this case, "If these
> schemas differ on assigning identity then you cause all sort of sticky
> issues."
>
>> It sounds really strange for me.
>
> Exactly because it is likely to cause problem.

As you know, the schema-blind middleware _must not_ apply
wrong schema to a document. If it does, the fact is a
serious accident by itself.

> Either
>   - use default YAML schema (previous emails) and then a "schema blind"
> "YAML middleware" processor might make sense, but who knows - this is
> up to developers of YAML middleware and or 'end user' applications,
>   - or use application specific schema, and you are very unlikely to be
> able to mix in "schema blind" YAML middleware.
>
> I think we all agree on these basics.

I partially agree. A middleware that can only work with
the default schema is almost useless for handling documents
with custom tags and custom tag resolutions regardless of
the identity issue.

But if we have a _schema-aware middleware_ and the end
user provides a correct schema to the middleware, everything
works fine.

Or, alternatively, schema-blind middleware can simply
provide some API that let end users to access directly to
the "partial representation graph of the document". (This
term is defined in the YAML spec.) With such APIs, end
users or applications can manipulate the document.
Although it will not be very easy for users, it is still
possible. My library also provides such APIs to let users
manipulate a YAML document that do not corresponds to any
known class objects.


>> IMO, any document can not stand without a valid schema.
>
> If we say "schema less document" is actually YAML document relying on
> "default schema" (with scalars working as users expect them etc), then
> I agree with you.

Yes, that is exactly what I meant.

> I like the "default schema" where "schema less" YAML documents work as
> I expect them to. I use YAML mainly with human (me) generated YAML
> files, and if the default schema were to change to require "scalars
> have identity" then the first thing I would require of any YAML tool I
> use, is that it support the "old style" schema.

Makes sense.

>> A schema-blind application must not apply any
>> possibly-wrong schema to any document. Namely, it can
>
> I think part of the problem is that we sort of lack a real world
> example of a practical for regular use "schema blind" tool. I think
> the examples of scalars needing identity have so far been a bit
> contrived - if '!not_float 3.14159' must have separate identity / be
> not equal to '!not_float 3.14159', that to my eyes looks like a
> decidedly application specific schema. I am still struggling to
> comprehend how that could be useful, even though I can certainly
> accept that it's possible it might be useful in some situation to
> someone.

Actually I do not think of any practical example for regular
use of schema-blind tool with documents with custom schema.

If the spec allows, I can serialize a class object, that is
provided with a builder function that accepts a single string
argument, into a scalar node as the users of my library expect.
Actually, in C#, many of the classes are provided with so-called
TypeConverter function that converts native class objects
from/to string. Then, the YAML documents look something like:

  - !Complex     3-2i
  - !ElementSize 14em,300pt
  - !MyClass     magical spell to generate the class object

I do not say my users always put importance on the identities
of such objects. But I can not say they will never do. It is
safer to keep the identity of the scalar nodes in this use
case.

I agree I can just forbid my users to store objects with
identities in this manner. But I think it helps nobody in
reality. It just makes the YAML document formally valid.


>> merely build partial representation graph and can not do
>> any more. If it go any further, it will very easily break
>> documents even without the identity issue, as partially
>> shown by my previous example with !!float tags.
>
> If an application requires such behaviour ("schema"), then it, of
> course, could not rely upon the default schema. This sounds sensible,
> yes?

Exactly.


>>> 3. YAML matches the native data structures
>>> <http://www.yaml.org/spec/1.2/spec.html#native data structure//>
>>> of agile languages.
>>
>> I agree that YAML nodes with standard tags match the
>> native data structures of agile languages pretty well.
>> But what about nodes with custom tags? How do they
>> matches the native data structures and how we can make
>> of the matching? Coud you show us some example use cases?
>
> How is that relevant to the discussion?
>
> Some languages will have layer 3/4 YAML constructors to support
> "arbitary" custom tags/ schemas - other languages might be a bit
> limited some how, although if such languages exist, then I think that
> language would be not very popular - we have relatively high "minimum
> standards" these days... I don't think BrainFuck is going to be at the
> top of the list for YAML implementations, although apparently BF -is-
> turing complete...

I hope I have shown a nice use case to use custom tags
to store _user-defined_ class objects in YAML. They are
native objects but not of _native_ data structures,
do they? Only in javascript, they may be?


>>> Which tend to make scalars immutable. Even Ruby is
>>> starting to see the error of its ways here so it is
>>> moving to make at least some strings be immutable
>>> (literals, "frozen", etc.).
>>
>> I repeat, I do not want to keep identities of !!str
>> nodes. If you are confident that the data indeed do not
>> need identity preservation, you can discard identities.
>>
>> My question is what we will gain by being allowed to
>> discard identities of nodes _with possibly unknown tags_.
>
> I am not a YAML library implementer/ programmer. That said, I think
> your question is too theoretical - if an application requires that any
> YAML middleware preserve scalar identity, then that application is
> naturally going to have to be pretty specific about which "YAML
> middleware" is allowed to be used in its processing chains.

Regardless of identity issue, you need a schema-aware
middleware to manipulate a document with tons of custom
tags and custom tag resolutions. If you call a schema-aware
middleware as "pretty specific", I agree with you.
But anyway, I do not think such a document is evil only
by the fact.


> In a decade or more, I have not seen the doors of this mailing list
> being broken down with questions such as "why is my YAML middleware
> not correctly processing my YAML communication channel between my YAML
> endpoints?"
> (Hint: we've never, ever, seen such a question.)
>
> Even a well-grounded "real world hypothetical" would move the
> discussion onwards at this point. But we don't even have a "real world
> hypothetical", let alone a real world problem.

Do you say I have invented a _new way_ of using custom tags?
I think the YAML spec writers intentionally designed YAML tag
system to allow such usage. Anyway, my library is already in
the real world and working pretty well. I will not be sad to
see none of schema-blind YAML tools can handle the YAML
documents that my library generates. If the users of my
application can read and write the generated document by their
eyes and hands, it is ok for me. IMO, the document is a nicely
written YAML document except for the identity issue.


>>> 4. YAML has a consistent model to support generic tools.
>>> that you
>>> I guess the question is, how strong is that consistent
>>> model? What does it _allow_ generic tools to do? The
>>> rules about scalar identity give generic tools the
>>> ability to do more than they could do if scalar identity
>>> had to be preserved.Note that a human with  a text
>>> editor can also be a generic,schema-blind tool.
>>
>> Ok, this seems a good example. So, what is additionally
>> allowed to the guy with a text editor by being allowed not
>> to preserve scalar identity?
>
> Common sense and "default expectation" - Linus Torvalds has a similar
> saying about kernel to user land interfaces, where POSIX has
> occasionally been allowed to be violated by linux, or a bug has been
> solved in one out of a number of ways, because of historical precedent
> and or user expectation.
>
> Scalars having identity breaks the expectations of the man with the
> editor. And if me editing a particular YAML document, do depend on
> scalar identity, I expect that that would require awareness of the
> schema for this document (which presumably I have, since I am manually
> editing the document).
>
> This is actually an example which favours the default expectation of
> users, which is, that scalar identity is not preserved, strings can be
> internet, etc.
>
> If a particular YAML "middleware" that I, the man with the editor, am
> using, let's say a YAML pretty printer, unexpectedly preserves scalar
> identiy, then I'll tell the dang pretty printer where to go.
>
> If I am programming some transactional multi layered software, and
> some middleware layer needs to work in a particular way (preserving
> identiy, or not preserving identiy), I'm simply going to make sure the
> YAML tool I use supports the mode of operation that I require.
>
> I think you might be trying to solve a non existent problem?

Besides the identity issue, a YAML document with custom tags
and custom tag resolutions are already surprising to those
who has limited knowledge on the schema. Identity preservation
of scalar node do not make it very much worse.

And I repeat, even you mistake the need for identity preservation,
it will not cause big trouble in reading/editing a YAML document
with text editor unless you intentionally alias or unalias the
nodes. If you know the meaning of the document well, you will
never do it. If you do not know the meaning of the document well,
you will not want to do it, neither, hopefully.


>>> 6. YAML is expressive and extensible.
>>>
>>> You could argue that not preserving scalar identity
>>> requiresa morecumbersome expression of some native
>>> data (e.g. wrappinga scalarin a collection "just
>>> because" you want to ensure itsidentity ispreserved -
>>> similarly to having to use an !!omapinstead of the
>>> cleaner map syntax for PHP dictionaries).
>>> "Everything" still _can_be expressed, though.
>
> This sounds intuitive to me.
>
>> If the benefit is larger than the labor, I will accept
>> the restriction.
>
> For starters, your !!not_float example seems artificial to me - like
> trying to solve a non existing problem. Changing an "intuitive to
> humans" aspect of the current YAML spec (/ default schema), in order
> to solve a non real world problem, would be a step backwards for YAML.

I hope the example above explains the problem.


>> For !!omap issue, I am about neutral.
>> I see it can prevent people unintentionally breaking a
>> YAML document by swapping the key order. I evaluate this
>> happens much more easily than someone aliases or unaliases
>> unexpected nodes.
>
> Map and OrderedMap are well defined (mathematically) concepts. YAML
> supports both, map by default schema, any other type of map by
> alternate schema/ tags. Any middleware will only ever be employed in a
> processing pipeline where it makes sense to use that. There is no real
> world problem that has knocked on the door.

I'm afraid I do not catch your point correctly.
Do you say people never accidentally swap key orders of
!!omap if they are familiar with the meaning of the document?
I almost agree with you then. In that sense I am about neutral.


>> On the other hand, if we see much more useful use cases
>> by allowing users to store key-order-aware hash or
>> key-duplication-unaware objects in mapping nodes, I
>> would be convinced to discard that restriction as Zanaan
>> is trying to.
>
> My goal has been trying to understand YAML on a deeper/ more precise
> level, so thank you for entertaining my slow understanding. My
> examples were put to help me understand your question, so I could
> understand YAML better - I am not trying to discard a restriction of
> YAML.
>
> Perhaps the question is: should the 'default schema' "hash map
> concept" be !omap or !map? As long as I know which the default schema
> is, I personally don't have attachment either way.
>
> In my initial tests of beginning to rewrite my little "learning YAML
> with Java beans" project without tags, all strings are "cookies" and
> all numbers are "values" - i.e. they have no identity, and if they
> did, that would be a problem for me - I would be immediately asking
> around for a library supporting "old/ original YAML" schema.
>
> For me, where say a string has to have identity, that would be because
> it represents a class name (Java bean name) - and of course, every
> other appearance of that exact sequence of characters (e.g. "T e s t B
> e a n" (without the spaces)), represents exactly the same entity/
> thing/ field/ class/ bean! A sane hypothetical where the opposite
> needs to be true, completely eludes me.

Preservation of identity of nodes with custom tags is not
likely to hurt you in reality. As I wrote, I'm not saying to
preserve identity of nodes with standard tags. Unless it is
required by tags, the middlewares will not always preserve
identities. I only say "let the tags determine discarding or
preserving identities." Unless the middleware has a good
support on the custom tags, it is nonsense to discus such
things because we have no real ways to do it anyway. Without
good middlewares, we will not be able to handle such documents
regardless of identity issue. So, it only matters when a
middleware provides any API for a custom schema to require
identity preservation for some tags. If this does not happen
around you unfortunately, the change of the spec will have
nothing to do with you.

The implementer of a YAML library has to think of identity
preservation only when it can accept custom schema.

Technically, it is easy to preserve the identity of scalar
nodes in their representation graph unless the library
intentionally discard the identity. Note that the scalar
nodes are not represented by native scalars in the graph.
They are almost always represented by some class objects
that naturally preserve their own identities.


>>> One of the reason we explicitly listed the goals,
>>> _in order_,was to break ties when different goals
>>> pushed us in differentdirections. Order the goals
>>> differently, and you'll get adifferent spec. I think
>>> you would end up with JSON if youorderedthem in a
>>> different way. Or even, god help us, XML ;-)
>>
>> I strongly agree with this statement and seemingly
>> the order itself is nice, too. But as I commented on
>> the first item, the order can not be super strict.
>
> Au contraire! I support a "super strict" order, so YAML design
> decisions are consistent over the years. This has been the case - one
> of the really nice things about YAML. Now we have "-layers-, onion
> boy" (with apologies to Shrek), so I'm even happier. Design
> consistency in YAML is truly awesome. And the order of design
> priorities is very appealing to me personally. Did I mention I like
> YAML?

Ok then, drop custom tags and custom tag resolutions
from the spec completely. Use only standard tags.
Use only standard schema.This will make YAML documents
more readable. Does It reduce the functionality of YAML?
Don't care. Functionality is given a lower priority.
If readability matters, forget about anything else.

If you say GO SUPER STRICT!, nobody can oppose this
decision.

I'm afraid you may indeed decide restricting the use
of custom tags and custom tag resolution. I understand
it can be one way to go, but...


>> We always have to optimize the balance of conflicting
>> ways to the goals.
>
> Of course, when a convincing argument can be put that a chosen order
> of priorities, or to pick a random example, rules applying in a
> particular logical layer of the system, ought be changed, then that
> makes for a great discussion.
>
>> Another point is, I feel that most of the YAML documents
>> in your mind are those that can be stored in JSON but
>> those in my mind are not always. YAML documents with
>> full of custom tags are not easily stored in JSON.
>
> I disagree. Every tag can be transformed into a two element list,
> where the first element is the tag, and the second element is the node
> content that was tagged. Or instead of a two element list, think a two
> element map, e.g.:
> -
>    tag: omap
>    content:
>      blah
>      blah

It may depend on personal standards.
I do not see this easy.


>> If we give up YAML, the next choice will indeed be XML.
>
> Nope. The next choice will be YAML 2. Then the next choice will be
> JSON 1. Then the next choice will be JSON 2. Then the next choice will
> be native serialization in your language of choice. Then custom binary
> serialization. Then a continually permutating algorithmic mixing
> stream, just for laughs. There are infinite alternatives to XML and if
> worse comes to worst, I suggest hard transcoding your data into COBOL
> statements and serializing those in Base63.
>
> Nowhere would one willingly choose XML.
>
> Hell, I'd choose HTML if I was ordered to use XML on a project, just
> to make sure the next guy who touches that code knows that the
> serialization format has to be changed.
>
> Manager: "Z, did you finish that, what's it called again? Yeah, the
> XML serialization?"
>
> Me: "Sure did! And it's Netscape 3.1 compatible too." <snigger>HTML<snigger>
>
> Manager: <used to work at Best Buy, does not understand>"Oh, cool!
> That's just great! I knew I'd finish on time, and I'll tell marketing
> our new name too, 'NewScope compatible' - has a great ring to it.
> Great ring! Good job Z, you'll go places you know. Go places in this
> world!"

I agree. The next choice for me will indeed be
_my own language_ that differs from YAML only at some
small differences: equality, identity and what else? ;)


I hope I have convinced you guys that at least some of
the benefits of allowing non-preservation of scalars with
custom tags are kind of imagination. Letting tags to
choose preservation of node identity will hurt you very
little if any. Letting tags to choose the way of evaluating
equality, too. Especially, if you have no middlewares that
understands custom schema around you, they have almost
nothing to do with you, anyway.


Best,
Osamu


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140