Re: YAML2

Oren Ben-Kiki wrote:

> Ok, that's a fair use case. Please add it to the Wiki.
> 
> It is pretty horrible that the OS allows such shenanigans, but I guess 
> that's a price we have to pay for building OS-es before Unicode existed.

I want to take SERIOUS objection to your idea that not code enforcing 
Unicode standards in the operating system kernel is "shenanigans".

Windows allows invalid UTF-16 in it's filenames, yet you don't hear 
people saying this is somehow "not supporting Unicode". This is a 
DISGUSTING double standard and I consider an insult to many people 
including the recently-deceased Ritchie.

The FIRST operating system to correctly handle Unicode was Plan9, more 
than a quarter century ago, and it explicitly uses bytes in all the 
apis, despite the fact that back then it was strongly believed that 
there would never be more than 2^16 Unicode code points. This is because 
it was written by people who were uninterested in scoring 
politically-correct points, and realized that even if ASCII had some 
"advantage" by being shorter, this actually was a benefit to all users 
of all languages by getting rid of "encodings" immediately. We still 
have "encodings" in Windows and Unix decades later, showing how correct 
they were.

> Nothing prevents you from adding a \x80 at the end of the file name in 
> the YAML file, using VI or Notepad or whatever. I bet that some YAML 
> libraries will even load it into a non-valid UTF-8 "string" in memory, 
> "illegal" though it may be.

No, the YAML 1 parser will expand "\x80" to the equivalent of "\xC2\x80" 
in my proposed YAML 2, which is not distinguishable from other sequences 
that are allowed in valid UTF-8. Therefore this will not work.

> But does this mean we need to mandate that all YAML implementations 
> silently create invalid UTF-8 strings in memory? That seems a bit 
> excessive...

To exactly work as I defined, you must use an internal format that can 
store all valid Unicode code points, plus 128 invalid UTF-8 bytes and 
2048 invalid UTF-16 words. For instance UTF-32 where the codes for the 
surrogate halves are used for invalid UTF-16, and 128 codes greater than 
0x10FFFF are used for invalid UTF-8.

You CANNOT use UTF-16 internally, because there is a back-compatibility 
requirement that two \uNNNN in a row that happen to form a valid 
high+low surrogate pair must turn into six UTF-8 bytes rather than 4. 
This rules out using UTF-16 as an intermediate form as there are not 
enough different code units to store this information and distinguish it 
from valid non-BMP characters.

The same argument may seem to rule out using UTF-8 internally, but I 
think it can be acceptable that the result of several \xNN that happen 
to be arranged as a valid UTF-8 character encoding is *undefined*. An 
implementation can return either a single Unicode character for this or 
N code units when UTF-16 is asked for. We could also require that the 
result be a single Unicode character, although that may *force* backends 
to use UTF-8 in any practical implementation, as you were worried about.

> If I had to deal with this use case, I'd use something like:
> 
>     filename: !badstr BadName\x80

This requires use of something other than backslash, as the YAML parser 
interprets the backslash in a lossy method by which this data cannot be 
recovered (you may think this is false, but you are thinking about 
"double encoding" which restricts the strings to ISO-8859-1).

'%' is often suggested, but then real '%' has to be quoted and this in 
fact makes the YAML definition much more complicated. Also this will 
force a huge amount of other (though potentially desirable) changes to 
YAML syntax to unify the URL and scalar rules.

My other complaint about any such suggestion is that any program 
outputting YAML will probably just put "!badstr" in front of EVERY 
instance of the strings, rather than doing a test, and thus defeating 
your proposed purpose. The test will still be to see if the % sequence 
is in the string, which is no different than looking if a \x is in the 
string.

------------------------------------------------------------------------------
Get your Android app more play: Bring it to the BlackBerry PlayBook 
in minutes. BlackBerry App World™ now supports Android™ Apps 
for the BlackBerry® PlayBook™. Discover just how easy and simple 
it is! http://p.sf.net/sfu/android-dev2dev