Re: Next YAML: drop equality definition

On Mon, Mar 7, 2016 at 7:42 AM, Andrey Somov <[email protected]> wrote:

> >Oren, if you are suggesting that {a: 1, a: 2} *must* be detected at the
> parser level, then I'd have to disagree from the streaming >standpoint,
> since the 2 keys may be lightyears away from each other.
>
> I completely support that {a: 1, a: 2} must NOT be detected at the parser
> level.
>

That's not the right wording. Oren said that the detection must be *allowed*,
and I agree.

I wrote that before I got to the part below where I reread what Oren said:

Really? People would be very surprised to hear that { a: 1, a: 2 } is
> actually OK because some application somewhere _may_ decide it wants scalar
> string keys to use "identity-based equality".
>

> So *No*.
>

> For increased readability and portability, the above _must_ be allowed to
> be flagged as a duplicate by a YAML processor regardless of what the
> application is. And _should_ be flagged so by "well behaved" processors.
> Even if they do _not_ have an application layer.
>

Whoops, I misread your sentence. "_must_ be allowed to be flagged". Yes I
completely agree with that statement. *If* a parser wants to detect
duplicate content/tag scalar keys it must be *allowed* to. I first read
this to say that parsers must. Since you and I have considered streaming
from the start, I was a bit surprised that you would say that. :)



> I must admit that I do not follow the discussion here because it is too
> much,
> but I agree with Osamu.
>

I don't follow you. You agree with Osamu on what?


>
> Cheers,
> Andrey
>
>
>
> On Mon, Mar 7, 2016 at 4:27 PM, Ingy dot Net <[email protected]> wrote:
>
>> Oren, Osamu,
>>
>> My concern with this thread is that you seem to be limiting your meaning
>> of "YAML processors" to Loader/Dumper processors (processors that carry
>> data from text to native programming in-memory objects, and back). ie Where
>> the "Application" level uses native objects.
>>
>> While that is certainly the normal case, YAML conjecture should also be
>> weighed against processors that stop after the parser (at the event
>> stream). ie Where the application is a streaming filter/mapper. ie No graph
>> construction or native objects ever happen.
>>
>> YAML conjecture should also be weighed against non-terminating (or
>> extremely huge) YAML nodes (documents, mappings, sequences).
>>
>> Oren, if you are suggesting that {a: 1, a: 2} *must* be detected at the
>> parser level, then I'd have to disagree from the streaming standpoint,
>> since the 2 keys may be lightyears away from each other.
>>
>> FWIW, libyaml doesn't reason on keys and shouldn't. libyaml makes a great
>> streaming parser/emitter.
>>
>> Key order can't really even be discussed at this level. Key order *can't* be
>> changed by a parser, so of course you can make it meaningful if you decide
>> to. You just have to be aware that your meaningfulness expires should your
>> YAML be used in a graphing context.
>>
>> More below:
>>
>> On Sun, Mar 6, 2016 at 10:27 AM, Oren Ben-Kiki <[email protected]> wrote:
>>
>>> On Sun, Mar 6, 2016 at 7:00 PM, Osamu TAKEUCHI <[email protected]> wrote:
>>>
>>>> Oren,
>>>>
>>>> I expect we share the same thinking that the
>>>> definition of equality belongs to the domain
>>>> specific data type, not to the serialization
>>>> language.
>>>
>>>
>>> Pretty much.
>>>
>>>
>>>> So, unless it makes the serialized
>>>> documents much more readable or portable, a
>>>> serialization language should not determine
>>>> its own equality or identity definition.
>>>>
>>>
>>> That's a bit "unless".
>>>
>>>
>>>> I agree duplicate key should be detected by
>>>> YAML processors because we do not want users
>>>> to use duplicate keys for overwriting the
>>>> values of predefined keys.
>>>
>>>
>> There is no concept of overwriting in a streaming processor. You need a
>> graph to do that.
>>
>>
>>>
>>> Yes.
>>>
>>>
>>>> The key order in
>>>> a YAML mapping should not have meaning.
>>>>
>>>
>>> Very strong yes.
>>>
>>
>> Depends on the layer.
>>
>>
>>>
>>>
>>>> But it can be done without defining equality
>>>> in YAML spec. YAML processor can use native
>>>> equality evaluator of the data at its
>>>> construction stage and it should do so.
>>>>
>>>
>>> No. You assume there _is_ a construction stage. There need not be one.
>>>
>>
>> Oren. I agree, although I think you mean that construction is skipped on
>> the way to native, where I'm saying that processing might never get to that
>> level.
>>
>>
>>>
>>>
>>>> If the layered structure of the YAML processor
>>>> do not allow it, the layered structure itself
>>>> should be revised. I don't see how the layered
>>>> structure is related to the current topic,
>>>> though.
>>>>
>>>
>>> It is crucial to the discussion. We all agree that key duplication
>>> detection _must_ be done at the application layer, but the point is that a
>>> _limited_ form of key duplication detection _may_ and _should_ be done,
>>> especially in YAML processors that do not even _have_ an application layer.
>>> This is because, as you put it, "it makes the serialized documents much
>>> more readable or portable".
>>>
>>
>> If the application is not a graph then duplication mustn't and likely
>> can't be done.
>>
>>
>>>
>>>
>>>> Similarly, I do not want to forbid PHP users
>>>> to store a PHP's native key-order-aware hash
>>>> into a key-order-unaware YAML mapping
>>>
>>>
>>> The problem is, how can you tell whether this is/not safe to do? When
>>> dumping such a hash table to YAML, the application needs to provide some
>>> hint to the YAML processor whether this is actually safe. By default, it is
>>> _not_ safe, so without an explicit hint, the YAML processor _should_ do the
>>> safe thing and emit it as an !!omap.
>>>
>>
>> PHP is not the outlier. Most JavaScript implementations have predictable
>> key order. I use JS all the time to preserve key order when doing things
>> like converting JSON to (block formatted) YAML.
>>
>>
>>>
>>>> Meaningfulness of the data identity should
>>>> also belong to the specific data types. As
>>>> shown by the previous examples, the difference
>>>> in the semantics of a scalar node and that of
>>>> a complex node is not always clear.
>>>
>>>
>>> Looks pretty clear to me. Scalars are "values". They have _no_ identity,
>>> they have _only_ content. Complex nodes have identity, and as you pointed
>>> out, this means their actual content may be irrelevant (for comparison).
>>> The current spec gets that last point wrong.
>>>
>>> I think the core issue here is identity of scalars. You seem to assume
>>> that a YAML processor _must_ preserve the identity of scalars. That is, it
>>> _must not_, for example, use interned strings for keys. The current spec
>>> says the opposite. A YAML processor _need not_ preserve scalar identity and
>>> it _may_ use interned strings and other similar tricks. It is definitely
>>> not required to keep the identity of, say, integer scalars!
>>>
>>> ... without declaring
>>>> possible non-preservation of identity for
>>>> scalars, nobody will think a data with an
>>>> identity-based equality evaluation must be
>>>> stored as a collection node and must not as a
>>>> scalar node. It brings some surprise to users.
>>>>
>>>
>>> Really? People would be very surprised to hear that { a: 1, a: 2 } is
>>> actually OK because some application somewhere _may_ decide it wants scalar
>>> string keys to use "identity-based equality".
>>>
>>> So *No*.
>>>
>>> For increased readability and portability, the above _must_ be allowed
>>> to be flagged as a duplicate by a YAML processor regardless of what the
>>> application is. And _should_ be flagged so by "well behaved" processors.
>>> Even if they do _not_ have an application layer.
>>>
>>
>> Whoops, I misread your sentence. "_must_ be allowed to be flagged". Yes I
>> completely agree with that statement. *If* a parser wants to detect
>> duplicate content/tag scalar keys it must be *allowed* to. I first read
>> this to say that parsers must. Since you and I have considered streaming
>> from the start, I was a bit surprised that you would say that. :)
>>
>> My other points stand.
>>
>> ...
>>
>>
>>
>>>
>>>> Such restriction will improve YAML's readability
>>>> and portability very little if any.
>>>
>>>
>>> We'll have to agree to disagree, I'm afraid...
>>>
>>>
>>>> Actually, I
>>>> believe the restriction is currently not known
>>>> widely and very few libraries and applications
>>>
>>> have ever utilized it.
>>>
>>>
>>> A pity.
>>>
>>>
>>>> I don't think many
>>>> existing YAML document loose its meaning if we
>>>> drop the restriction.
>>>
>>>
>>> No valid YAML documents will, that's for sure ;-) But that's besides the
>>> point.
>>>
>>>
>>>> So, let's make the spec simpler by dropping the
>>>> definition of YAML's own equality and identity
>>>> preservation.
>>>>
>>>
>>> There's no such thing as not addressing the issue of identity and
>>> equality in the spec. Either you _require_ a YAML processor to preserve the
>>> identity of scalars (including, horribly, simple integers), or you do not.
>>> Either way it needs to be stated in the spec.
>>>
>>> We chose to say a YAML processer _need not_ preserve the identity of
>>> scalars. Given this, then an application _must not_ use scalar identity for
>>> equality comparisons. Given this, then _regardless_ of the application's
>>> definition of equality, we can predict with 100% certainty that { a: 1, a:
>>> 2 } contains a duplicate key.
>>>
>>> What I want to say in the spec is:
>>>> A well-behaved processor _should_ detect a
>>>> duplicate key and flag it as an error if it
>>>> can correctly evaluate equality of nodes.
>>>>
>>>
>>> So far so good.
>>>
>>>
>>>> It _must_ aware that a data with some specific
>>>> tags may have some custom comparison algorithms,
>>>> including the one based on the data identity.
>>>>
>>>
>>> Yes, the current spec gets the identity point wrong.
>>>
>>>
>>>> Namely, two YAML nodes of same values and same
>>>> tags can be evaluated to be unequal by an
>>>> identity-based evaluator,
>>>
>>>
>>> _Only_ if these are complex nodes.
>>>
>>>
>>>> while two YAML nodes
>>>> of different values and even different tags can
>>>> be evaluated to be equal by some specific
>>>> evaluators. Note that javascript do not
>>>> natively distinguish an integer 0x01 with a
>>>> sequence [1] as mapping keys.
>>>>
>>>
>>> You keep conflating false positives with false negatives. False
>>> negatives are _fine_. It is OK for the processor to miss some cases of key
>>> duplication. In fact it is expected. The application is the final arbiter
>>> of key equality. You can keep on piling as many examples of "the processor
>>> can't detect keys in <some example> as duplicated" as you want. OF COURSE
>>> there are such cases.
>>>
>>> But this does not mean in any way shape or form that we allow false
>>> positives. A YAML processor must not ever complain about key duplication
>>> when such a duplication does not exist. Now, in JavaScript, PHP, Perl,
>>> Ruby, Python, C++, and any other valid YAML system, { a: 1, a: 2 } _does_
>>> have a duplicate key. So a processor _is_ allowed and _should_ complain
>>> about this, _regardless_ of the application-defined equality operator.
>>>
>>> It is also warned that tags of nodes can be
>>>> implicitly specified by the path of the node
>>>> from the root. So, a schema-blind YAML processor
>>>> can never know how to resolve a tag for any
>>>> tag-unspecified node.
>>>
>>>
>>> The path to both the "a" keys in { a: 1, a: 2 } is, by definition, the
>>> same (the path to all keys in the same mapping is, by definition,
>>> identical). So whatever tag is assigned to one of them, by definition, the
>>> same tag must be assigned to the other as well. The application _can't_ use
>>> different tags to distinguish between them. It _can't_ use their identity
>>> to distinguish between them because the YAML processor need not give them
>>> different identities. It _can't_ use their content to distinguish between
>>> them because they have the same content. So, the application _must_
>>> consider them equal - there's just no other possible choice.
>>>
>>> So we _can_ complain about them being equal at an earlier processing
>>> stage. We do not _require_ a YAML processor to do so, but we _allow_ and
>>> _encourage_ it to do so.
>>>
>>>
>>>> A well-behaved YAML
>>>> processor _must_ be schema aware,
>>>
>>>
>>> Now this is just plain wrong. YamlReference is a YAML processor. It
>>> implements the parsing stage. It has no clue whatsoever what schema is
>>> used. Schema-blind YAML processing is, for me, an important use case.
>>>
>>> And if every possible schema in the universe _must_ decree that two keys
>>> are equal, then we don't need to know the _specific_ schema, because
>>> whatever it is, it will also _have_ to declare them equal.
>>>
>>> Oren.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Yaml-core mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/yaml-core
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://makebettercode.com/inteldaal-eval
>> _______________________________________________
>> Yaml-core mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/yaml-core
>>
>>
>
>
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://makebettercode.com/inteldaal-eval
> _______________________________________________
> Yaml-core mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/yaml-core
>
>

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://makebettercode.com/inteldaal-eval

_______________________________________________
Yaml-core mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/yaml-core