Draft 2008-04-06

Is done and has been uploaded! It has cost me the spring vacation but it
has been three years of "not getting around to do it", so...

Clark will need to install it in the http://yaml.org/spec/cvs directory;
currently it is available in
http://ben-kiki.org/oren/YamlReference/spec/current{.html,.ps,.pdf}.

It is accompanied by version 0.9 of the YamlReference implementation,
available at http://ben-kiki.org/oren/YamlReference and of course the
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/YamlReference
package database entry. It is also running the current
http://dev.yaml.org/ypaste version so feel free to play with it.

I have tried to incorporate all the spec fixes people have sent to the
list in the last three years. Kirill's site was also very helpful
(thanks Kirill!). I have lost track at some point of all the people and
corrections (there were a lot of e-mails, on and off the list). My
apologies to everyone for me taking so long to do this, and my thanks
for your effort in reviewing the old spec!

The new spec uses the improved YamlReference productions, which are
somewhat simpler than the old set. They also have the advantage of being
testable using YamlReference and ypaste, so I am much more confident
they are "correct".

I have also revamped the structure (and to some degree the wording) of
the syntax chapters. I believe the result is (at least somewhat) less
opaque than the previous spec. That said, I found Tim Hochberg
observation to be very apt:

> YAML has always seemed precariously balanced between user friendliness 
> and chaos.

I think this *perfectly* expresses YAML and the inherent complexity of
the spec. We really should work this sentence into the spec
somewhere :-)

So, whether you are a YAML library developer, casual user, or a would-be
language lawyer - please give this a look, try stuff out in ypaste, and
post your comments to this list!

My intention is to do a second pass of corrections by the end of the
month and assuming nothing major comes up, finalize the result as "the"
YAML 1.1 spec. We could then focus our efforts in bringing the
implementations to conformance, the type repository (which also needs a
lot of work), and so on. This is all subject to Clark's OK of course.

There are several categories of comments people can make, and it would
be helpful if they would clarify which one they have in mind:

Omission:

The spec does not cover some edge case. I doubt we'll see any of those.
Then again these are the hardest issues to find... so most cookies for
these ones :-)

Language:

There is a better way to express some point (or I made a typo). English
isn't my native language and I am not a professional writer so I have no
illusions about the quality of my writing :-) If you have a better idea
on how to convey something, I'll be more than happy to incorporate it.

Bug:

The spec contradicts itself. I hope I have eliminated all(most) of
these. Note the productions are technically ambiguous. The syntax
"works" by relying on productions being "greedy". This is still a valid
formal definition... just barely so :-)

Error:

The spec describes behavior which is different than the consensus we
have reached on this list. The best way to demonstrate this is using
ypaste - executable formal specifications are fun!

Feature:

The spec describes behavior which is different than the consensus we
"should" have reached on this list :-) That is, a change request. In
general these will be rejected unless they have a very good use case
they resolve, have low implementation cost, do not harm existing
documents, have negligible downside, and are in general considered to be
a good idea by almost everyone. We had 3 years to collect such ideas
after all...

Here is a list of the "Features" that were added to this spec (compared
to the 3 year old version):

- JSON compatibility. {"a":12} is a mapping and http://example.com is a
plain scalar. This is the main new feature in this spec.

- Tag resolution. Applying a ! tag to any node "disables" the "implicit
typing". This works for plain scalars, as well as for all mappings and
sequences (so ! { x: 1, y: 2 } is a vanilla mapping/hash and not a
"Point"). Quoted/block scalars are already "disabled" (are interpreted
as strings). Note this is all by convention and under complete
application control.

- Anchor names are restricted not to contain , [ ] { } characters to
avoid confusion inside flow collections. That is, [&a,1] is the same as
[ &a , 1 ] as people expect.

- The stream format is "sloppy", allowing for end markers to be repeated
etc., for easy appending and concatenation of streams.

Here are examples of features that we already discussed ad nauseum and
did not make the cut. These are huge potential black holes of
conversations, so please let us not re-open them:

- http://example.com is a plain scalar and not a key: value pair. So
{a:12} is a set with one entry 'a:12'. Write {a: 12} or {"a":12}.

- Plain scalars allow , [ ] { } in block collections but disallow them
in flow collections (and keys) to avoid ambiguity.

- Types other than string, mapping and sequence are out of scope of the
spec. Yes, the repository is woefully out of date. Lets get the spec out
the door and then fix the repository.

- Defining a canonical format is out of scope. The new spec does make
the suggestion we define a YSON format that can serve this purpose (and
also be a stepping stone between JSON and YAML). Again, something we can
devote effort for after we nail the spec.

Here is a sample feature idea that we did *not* discuss and we just
might be able to squeeze in at the last second. If enough people like
it. Maybe. Comments about it are of course welcome.

---

Title: Restricted Tag Character Set

Description:
 Disallow , [ ] { } in tags (similar to disallowing them in anchors
 which we already do) to minimize the potential confusion when tags
 are used in flow collections. Note that if such characters must be
 in a tag (for some obscure reason), it is possible to escape them
 using %xx.

Use case:
 Writing { !foo,[] } is interpreted today as { !<!foo,[]> "" }. Under
 the above proposal it would be { !<!foo> "", [ ] } which is what people
 expect.

Implementation effort:
 This can be viewed as the natural extension of the current rule that
 forces '!' to be escaped as %21 in certain cases (see example 6.26 in
 the spec). The proposal would make ns-tag-char to be more restrictive
 (will also exclude the , [ ] { } from it) and then use ns-tag-char
 instead of ns-uri-char everywhere (the TAG directive could still use
 the full set). If anything, this simplifies the  implementation (and
 the spec).

Effect on current documents:
 AFAIK the language-specific tags generated by Ruby/Perl/etc. do not
 use ! , { } [ ] in their tags. No sane human would use them either. So
 it seems that there would be no effect.

Downsides:
 No obvious ones so far.

...

Thanks again to everyone for investing all the effort in bringing the
spec to where it is today!

"Once more into the breach"

	Oren Ben-Kiki


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Register now and save $200. Hurry, offer ends at 11:59 p.m., 
Monday, April 7! Use priority code J8TLD2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone