Re: YAML2

The YAML parser will only expand \x80 if it was inside double quotes. My
example did not use double quotes:

  foo: !badstr BadName\x80

Which is different from:

  foo: !badstr "BadName\x80"

In the former case, the scalar is passed as-is (well, subject to plain
scalar processing such as folding, which do not include expanding escape
codes) to the application - specifically, by using whatever rules the
badstr tag has set up. Just like the int tag knows how to convert the
characters 1 0 into an integer with the decimal value 10, a badstr tag
would know to convert the characters \ x 8 0 to a single byte with
hexadecimal value 0x80. In both these cases, this isn't done by the YAML
parser itself. Escape sequences inside double quotes are handled by the
YAML parser itself, outside the application tag control.

It is interesting that Plan9 also does not enforce validity. FWIW, I think
that Windows allowing invalid UTF-16 in file names is a travesty, but
that's just me (I also think that windows should have made the space
character equivalent to underscore in file names, the same way that it
makes capital and lower case letters equivalent, so I could easily type in
/.../program_files/... in a command line argument. But I digress). This is
neither here nor there.

As you correctly point out, most systems use an actual "must be valid str"
internal representation for the data type "string"; they tend to throw
exceptions and so on if seeing stuff like invalid high+low surrogate pairs
and so on. !!str is meant to be loaded into exactly that - simple UTF-16 or
UTF-8 internal "string" representation, which is rather fussy about
validity in many systems. As you point out, for your use case you need a
different internal representation that can handle all valid Unicode data
point + some invalid ones. I suggested (a bit tongue-in-cheek) to tag this
other, not-quite-a-normal-string internal representation "badstr". A better
name would be "binstr" - A string that also contains "binary data"
(arbitrary bytes).

Using an explicit binstr tag indicates to the YAML parser to "please load
this into a String-like internal data type that also supports the
additional invalid code points". This may or may not be the same internal
data type that normal strings are loaded into - depending on how fussy your
system's string data type is. Perhaps in Plan9, these might be the same
internal data type, while in Java they might be different data types.

If we allowed all normal strings to contain arbitrary bytes, this means
that we wouldn't be able to load them into the normal fussy-about-validity
internal string type. We would be forced to load everything into the
less-usual, not-what-you'd-expect less-fussy string-like data type; this
isn't a defensible design choice for languages whose standard string type
is fussy about validity - YAML strings would lose interoperability with the
rest of the system.

On output, I assume that an application would strive to use !!str rather
than !!binstr as much as possible. But that's really an application
decision, and not the YAML library's choice. E.g., a hypothetical
Windows-file-names dumping application would only resort to emitting
!!binstr if the filename was, indeed, a string-with-weird-characters as
opposed to a normal string. This would allow it to be safely loaded into
(say) a Java application without causing an exception, while minimizing the
pain to the human YAML reader.,

Have fun,

    Oren Ben-Kiki

------------------------------------------------------------------------------
Get your Android app more play: Bring it to the BlackBerry PlayBook 
in minutes. BlackBerry App World™ now supports Android™ Apps 
for the BlackBerry® PlayBook™. Discover just how easy and simple 
it is! http://p.sf.net/sfu/android-dev2dev

_______________________________________________
Yaml-core mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/yaml-core