Re: New invalid UTF-8 proposal using tags and %nn

On Sat, 2009-09-05 at 12:22 -0400, BlueGM wrote:
> So, to move things along (and you'll please forgive me if I overstep myself
> a bit again), I think the three questions that need to be asked are:

> 1) How do we define type of data that the tag represents? Most of the
> discussion has been about this, so... should be straight forward enough to
> answer.

Clear enough; the data is a byte array (same as for !!binary), with the
added note that it is expected to be "mostly" UTF-8 text with the
"occasional" arbitrary non-UTF-8 bytes.

> 2) What is the canonical form used for that data? Key to how YAML works with
> different data types.

Good question.The simplest way to define it is as (%[0-9a-z][0-9a-z])*.
Nobody said the canonical form had to be short or human-readable or
anything like that. It just needs to be unique. BTW, this is similar to
how the canonical form of !!binary is defined.

> 3) What are the possible presentation forms of that data? We might also
> suggest a preferred presentation or even comment that writers should not use
> this tag for strings that are entirely valid Unicode.

Basically any sequence of %nn bytes that happens to be the valid UTF-8
encoding of some valid Unicode character may be presented using that
Unicode character, except for '%' which must remain '%25' for obvious
reasons. From there on the normal YAML presentation rules take over,
e.g. inside double-quote strings, the Unicode character may be presented
as an escape sequence (\x, \u or \U). So for example it would be valid
(though extremely silly) to present the byte '%' as "\x252\x34".

> And, of course, we need to settle on a name, but we seem to be coming to the
> conclusion of that :-).

Yes, !!utf8u seems like the best choice all around.

> After these questions are answered, then a
> discussion of how PyYAML or some other implementation should expose this
> data might be worth having. Myself, I don't care how PyYAML will do so,
> which is why I'm trying to get off that topic.

I'll leave that to the PyYAML maintainers.

> That, and I'm eager to see
> the new type's definition.

Right. Another item to the YAML spec-related work TODO list :-S

> If we do continue discussing the UTF-16 (utf16u or whatever), then we'd need
> to answer the same questions for that type as well. I would suggest using
> the term "word" instead of "bytes" though (along with a definition of "word"
> since it varies based on the environment).

Definitely. But there doesn't seem to be a pressing need to define such
a type at this point.

Have fun,

    Oren Ben-Kiki



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july