Re: Types, byte codes etc.

Here is my take on the events/format. The main decision we need to make
is whether we define the events to be 1-1 with the input lines, or
whether we use a richer set of events to fully describe the nested data
structure - that is, whether we allow for implicit start/end events. 

Ingy's post assumed we do the latter, and this makes sense as far as an
events API is concerned. This complicates the mapping between the lines
and the events. It requires the mapping to maintain a state, keeping
track whether, at any given point, a stream, document and/or node has
"started":

    STREAM_START:
        Implicit, if stream is not "started" and any other event is
about to be fired.

    STREAM_END:
        Implicit, if stream is "started", reached the end of the input,
and no other events can be fired.

    DOCUMENT_START:
        Implicit, if stream is "started", no document is "started", and
any event is about to be fired.

    DOCUMENT_END:
        Implicit if document is "started", and reached the end of the
input, and no other events other than STREAM_END can be fired;
        ...  which must be followed by another document.

    NODE_START:
        Implicit, if stream and document are "started", no node is
"started", and any event is about to be fired other than DOCUMENT_END,
MAPPING_END or SEQUENCE_END.

    NODE_END:
        , if in collection, and then must be followed by another node;
        Or implicit, if a node is "started" previous event was SCALAR,
ALIAS or NULL;
        Or implicit, if a node is "started" and last fired event was
MAPPING_END or SEQUENCE_END,

Everything else is a simple 1-1 mapping from input lines to events,
based on the 1st character of the line:

    ANCHOR:
        Explicit &alias

    TAG:
        Explicit !<verbatim-tag>

There are two problems here, a minor one and a more serious one.

The minor one is that we want each of these to be on a separate line. My
latest YamlReference disallows that, although this is allowed in the
current spec. I need to go over the syntax and see how I can
disambiguate this back towards the current spec, at least for flow
nodes. Otherwise, like ingy posted, properties would have to prefix the
node value line, which would make the format much less "byte-coded
lines".

The serious one is how to handle tags when none were given in the
original input. The YAML spec requires one of the non-specific tags for
untagged scalars, ? for plain and ! for all others. This means that
naively, the following:

    ---
    - 1
    - "a"
    ...

Would be reported as:

    [
    !<?>
    "1"
    ,
    !<!>
    "a"
    ]

This of course kills JSON compatibility. Damn. The only way I see to
restore JSON compatibility is to omit the TAG event if none was
specified in the input, and have _two_ scalar values (plain and
double-quoted). Luckily for us there's no ambiguity, because of our
restrictions on plain scalars. So the above would have the canonical
byte code of:

   [
   1
   ,
   "a"
   ]

So, simple node values:

    PLAIN:
        unquoted

    QUOTED:
        "quoted"

    ALIAS:
        Explicit *anchor

A nasty problem with the above is the following:

   ---
   -
   ...

We can't express it as:

    [
    ]

Or as:

    [
    
    ]

Or as:

    [
    !<?>
    ""
    ]

So we have to hard-wire into our byte code the convention that a plain ~
is equivalent to a "no value":

    [
    ~
    ]

Another price we have to pay for JSON compatibility...

So much for the hard part. Collection node values are easy:

    SEQUENCE_START:
        [

    SEQUENCE_END:
        ]

    MAPPING_START:
        {

    MAPPING_END:
        }

    MAPPING_VALUE:
        :

For JSON compatibility, we must forego the use of an explicit
MAPPING_KEY ? line. We may want to add implicit events:

    PAIR_START:
        implicit, if in mapping value, not following MAPPING_VALUE, and
about to fire NODE_START.

    PAIR_END:
        implicit, if in mapping value, just fired NODE_END, and next
event is not MAPPING_VALUE.

And maybe also (for completeness):

    MAPPING_KEY:
        implicit, if in mapping value, between PAIR_START and
NODE_START.

Also remember that a ',' ends a node in collections, as described above.

Finally we _may_ also want to allow for directives, just in case.:

    DIRECTIVE:
        %directive arg1 arg2 ...

So, the result isn't quite as elegant as we could have hoped for, but it
isn't too bad. Lets see how we are doing per our goals:

* One (non-implicit) atomic event per line, indicated by the 1st character.
    Check, except for:
    Directive which is a complex event (with arguments). If we allow for it. No big tragedy here.
    PLAIN is detected by virtue of its 1st character not matching any of the other events types. Acceptable.

> * Use exactly zero or one lines per event.
    Check, except for the problem with tags/anchors (which I'll try and resolve).

> * Make it be valid JSON when possible.
    Check. This means no '---' header for 1st document; instead we use '...' to separate documents.

> * Make it be very clear and explicit otherwise.
    Check.

> The differences between YAML and JSON are roughly:

    Aliases, anchors, null nodes, tags, allowing any node type to be
used as a key, allowing multiple documents in a stream, and (maybe)
directives; and if none of these are used, the result is valid JSON.

Have fun,

    Oren Ben-Kiki

P.S. This weekend I had a huge emergency at work (many people working
over the weekend, with little or no sleep :-( So, I didn't have time to
do any of the things I wanted to (set up the SVN, fix the YamlReference,
etc.). I'll try again next weekend.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/