Re: Invalid UTF-8

On Wed, 2009-08-26 at 12:39 -0700, William Spitzak wrote:
> Oren Ben-Kiki wrote:
> 
> > I have no special interest in "helping Unicode". All I want is the
> > security of knowing that a YAML file is portable across systems,
> 
> My files contain the valid UTF-8 encodings of '\', 'X', and two hex 
> digits. They are valid Unicode and are therefore "portable across systems".

"True, but mostly irrelevant". If I try to load them using a YAML
library that uses UTF-16 (e.g., a Java one), and the \X escape sequence
specifies an invalid UTF-8 byte sequence, then the library will be
forced to throw an exception. So the file is not "portable" to such a
system (cannot be loaded by it).

As for the filename issue, you correctly said:

> I fail to see how libyaml's rejection of a certain subset of byte
arrays 
> just because it MAY be a set a filesystem will reject is any kind of 
> excuse however.

Fair enough. YAML really doesn't care about filesystems and the whole
issue is secondary, so lets drop it. For those interested :-)
http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html contains
a nice long rant about UNIX file systems, what POSIX does and does not
say, which OS supports what and so on (with links to lots of interesting
resources, standards, etc.). The bottom line is that POSIX defines the
"portable character set" for filenames to be basically ASCII letters,
numbers and the like; and that Plan9 (ever since 1998) actually enforced
this. But, again, this really isn't relevant for YAML.

What is relevant for YAML is the following property: If a YAML file can
be loaded by a conforming YAML library on platform X, it is also
guaranteed that the same YAML file can be loaded by a conforming YAML
library on platform Y. Alas, the "\Xnn" escape sequence does not satisfy
this; a file with certain "\Xnn" escape sequences can be read by a YAML
library on a UTF-8 platform but can _not_ be read by a YAML library on a
UTF-16 platform.

> They [Java] either need a 
> new byte-oriented api to the filesystem, or (more likely) a lossless 
> method to encrypt UTF-8 into UTF-16 arrays (a popular one is to do 
> UTF-16 "decoding" but turn invalid bytes into a 0xDCxx word and consider 
> the UTF-8 encodings of U+DC80..U+DCFF to be invalid byte sequences).

Now we are getting somewhere... so basically you are converting between
invalid UTF-8 byte sequences to (some) invalid UTF-16 byte sequences in
a lossless fashion; presumably it is possible to expand this to use
invalid UTF-32 byte sequences as well. Such a system could in theory be
supported by YAML, because it would allow a library to manage to load
the file regardless of the encoding used.

However before rushing into such a thing, I'd like to see a bit more
"baked" definition of how this works. For example, this handles
arbitrary invalid UTF-8 bytes, but does it also handle arbitrary invalid
UTF-16 bytes? What about invalid UTF-32 bytes? If someone came up with a
lossless way to convert arbitrary byte streams between all three
encodings, that would be pretty cool! 

I would not object _in principle_ for extending YAML in that direction,
_given_ such a mechanism became somewhat mature/acceptable (e.g., a
clear definition somewhere, adoption by some tools such as Java, iconv,
Unicode processing libraries), etc. I'd be _very_ wary of blazing the
trail here.

In the meanwhile, this is definitely out of scope for YAML 1.2. We have
no plans to start working on YAML 1.3/2.0 any time soon; we'd rather
focus the efforts in bringing existing implementations up to date with
the 1.2 spec, as this will solve much more burning problems for YAML
users today (most painful being JSON compatibility).

Have fun,

    Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july