Re: Next YAML: drop equality definition

On Sun, Mar 6, 2016 at 7:00 PM, Osamu TAKEUCHI <[email protected]> wrote:

> Oren,
>
> I expect we share the same thinking that the
> definition of equality belongs to the domain
> specific data type, not to the serialization
> language.


Pretty much.


> So, unless it makes the serialized
> documents much more readable or portable, a
> serialization language should not determine
> its own equality or identity definition.
>

That's a bit "unless".


> I agree duplicate key should be detected by
> YAML processors because we do not want users
> to use duplicate keys for overwriting the
> values of predefined keys.


Yes.


> The key order in
> a YAML mapping should not have meaning.
>

Very strong yes.


> But it can be done without defining equality
> in YAML spec. YAML processor can use native
> equality evaluator of the data at its
> construction stage and it should do so.
>

No. You assume there _is_ a construction stage. There need not be one.


> If the layered structure of the YAML processor
> do not allow it, the layered structure itself
> should be revised. I don't see how the layered
> structure is related to the current topic,
> though.
>

It is crucial to the discussion. We all agree that key duplication
detection _must_ be done at the application layer, but the point is that a
_limited_ form of key duplication detection _may_ and _should_ be done,
especially in YAML processors that do not even _have_ an application layer.
This is because, as you put it, "it makes the serialized documents much
more readable or portable".


> Similarly, I do not want to forbid PHP users
> to store a PHP's native key-order-aware hash
> into a key-order-unaware YAML mapping


The problem is, how can you tell whether this is/not safe to do? When
dumping such a hash table to YAML, the application needs to provide some
hint to the YAML processor whether this is actually safe. By default, it is
_not_ safe, so without an explicit hint, the YAML processor _should_ do the
safe thing and emit it as an !!omap.


> Meaningfulness of the data identity should
> also belong to the specific data types. As
> shown by the previous examples, the difference
> in the semantics of a scalar node and that of
> a complex node is not always clear.


Looks pretty clear to me. Scalars are "values". They have _no_ identity,
they have _only_ content. Complex nodes have identity, and as you pointed
out, this means their actual content may be irrelevant (for comparison).
The current spec gets that last point wrong.

I think the core issue here is identity of scalars. You seem to assume that
a YAML processor _must_ preserve the identity of scalars. That is, it _must
not_, for example, use interned strings for keys. The current spec says the
opposite. A YAML processor _need not_ preserve scalar identity and it _may_
use interned strings and other similar tricks. It is definitely not
required to keep the identity of, say, integer scalars!

... without declaring
> possible non-preservation of identity for
> scalars, nobody will think a data with an
> identity-based equality evaluation must be
> stored as a collection node and must not as a
> scalar node. It brings some surprise to users.
>

Really? People would be very surprised to hear that { a: 1, a: 2 } is
actually OK because some application somewhere _may_ decide it wants scalar
string keys to use "identity-based equality".

So *No*.

For increased readability and portability, the above _must_ be allowed to
be flagged as a duplicate by a YAML processor regardless of what the
application is. And _should_ be flagged so by "well behaved" processors.
Even if they do _not_ have an application layer.


> Such restriction will improve YAML's readability
> and portability very little if any.


We'll have to agree to disagree, I'm afraid...


> Actually, I
> believe the restriction is currently not known
> widely and very few libraries and applications

have ever utilized it.


A pity.


> I don't think many
> existing YAML document loose its meaning if we
> drop the restriction.


No valid YAML documents will, that's for sure ;-) But that's besides the
point.


> So, let's make the spec simpler by dropping the
> definition of YAML's own equality and identity
> preservation.
>

There's no such thing as not addressing the issue of identity and equality
in the spec. Either you _require_ a YAML processor to preserve the identity
of scalars (including, horribly, simple integers), or you do not. Either
way it needs to be stated in the spec.

We chose to say a YAML processer _need not_ preserve the identity of
scalars. Given this, then an application _must not_ use scalar identity for
equality comparisons. Given this, then _regardless_ of the application's
definition of equality, we can predict with 100% certainty that { a: 1, a:
2 } contains a duplicate key.

What I want to say in the spec is:
> A well-behaved processor _should_ detect a
> duplicate key and flag it as an error if it
> can correctly evaluate equality of nodes.
>

So far so good.


> It _must_ aware that a data with some specific
> tags may have some custom comparison algorithms,
> including the one based on the data identity.
>

Yes, the current spec gets the identity point wrong.


> Namely, two YAML nodes of same values and same
> tags can be evaluated to be unequal by an
> identity-based evaluator,


_Only_ if these are complex nodes.


> while two YAML nodes
> of different values and even different tags can
> be evaluated to be equal by some specific
> evaluators. Note that javascript do not
> natively distinguish an integer 0x01 with a
> sequence [1] as mapping keys.
>

You keep conflating false positives with false negatives. False negatives
are _fine_. It is OK for the processor to miss some cases of key
duplication. In fact it is expected. The application is the final arbiter
of key equality. You can keep on piling as many examples of "the processor
can't detect keys in <some example> as duplicated" as you want. OF COURSE
there are such cases.

But this does not mean in any way shape or form that we allow false
positives. A YAML processor must not ever complain about key duplication
when such a duplication does not exist. Now, in JavaScript, PHP, Perl,
Ruby, Python, C++, and any other valid YAML system, { a: 1, a: 2 } _does_
have a duplicate key. So a processor _is_ allowed and _should_ complain
about this, _regardless_ of the application-defined equality operator.

It is also warned that tags of nodes can be
> implicitly specified by the path of the node
> from the root. So, a schema-blind YAML processor
> can never know how to resolve a tag for any
> tag-unspecified node.


The path to both the "a" keys in { a: 1, a: 2 } is, by definition, the same
(the path to all keys in the same mapping is, by definition, identical). So
whatever tag is assigned to one of them, by definition, the same tag must
be assigned to the other as well. The application _can't_ use different
tags to distinguish between them. It _can't_ use their identity to
distinguish between them because the YAML processor need not give them
different identities. It _can't_ use their content to distinguish between
them because they have the same content. So, the application _must_
consider them equal - there's just no other possible choice.

So we _can_ complain about them being equal at an earlier processing stage.
We do not _require_ a YAML processor to do so, but we _allow_ and
_encourage_ it to do so.


> A well-behaved YAML
> processor _must_ be schema aware,


Now this is just plain wrong. YamlReference is a YAML processor. It
implements the parsing stage. It has no clue whatsoever what schema is
used. Schema-blind YAML processing is, for me, an important use case.

And if every possible schema in the universe _must_ decree that two keys
are equal, then we don't need to know the _specific_ schema, because
whatever it is, it will also _have_ to declare them equal.

Oren.

------------------------------------------------------------------------------

_______________________________________________
Yaml-core mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/yaml-core