Re: Invalid UTF-8

On Tue, 2009-08-18 at 19:31 -0700, William Spitzak wrote:
> I'm sure this has come up before, but anyway:
> 
> In our software it looks like YAML would be a great way to store are 
> retrieve data. However we have the need to store arbitrary bytes, with 
> the caveat that they are *LIKELY* to be UTF-8. Examples are Unix 
> filenames, and metadata stored in image files such as comments.
> 
> This requires that we be able to losslessly store invalid UTF-8.

I'm pretty uncomfortable with this. Ideally, text strings should be used
for text strings - which means valid Unicode characters. Binary data can
use something like !!binary and base64 encoding.

There are two possible separate options for  support for invalid
Unicode. The first is to allow it _directly_ in the file. I believe this
is what you meant when you wrote:

> 3. Mismatched surrogate pairs (ie invalid UTF-16) are accepted and read 
> and written directly. This means that a UTF-16 file read/written by YAML 
> can contain raw invalid UTF-16 sequences. The obvious encoding of 
> invalid UTF-16 to UTF-8 is used to read/write UTF-8 files. Notice that 
> this encoding is lossless and does not interfere with correct UTF-8 
> encoding of Unicode characters >= U+10000.

IMO this is a bad idea. It means one would no longer be safe opening a YAML
file in a text editor. Or be able to print it. Or run it through _any_
Unicode aware application. In short, this directly contradicts YAML's
primary goal of human-readable files.

The second level is to allow _escaped_ invalid Unicode characters in strings.
In effect, this converts double-quoted strings to binary blobs; instead of
using base64, they are using \Xnn notation for non-printable data.
The YAML file itself, however, remains a valid UTF-whatever file. I believe
this is what you meant when you wrote:

> 1. Invalid UTF-8 is stored by writing each erroneous byte using the new 
> escape sequence "\XNN" in quoted strings. Thus the saved files are 
> themselves valid UTF-8 and writing UTF-16 files still works.

While this is not _obviously_ a bad idea, I am still against it. It
means that trying to load such files into a Unicode-aware programming
languages may fail. This directly contradicts the second and third YAML
goals (match native data structures of agile languages; portable between
programming languages).

I can understand the temptation... but IMO making this sort of feature
part of the standard is not a good idea. FWIW, even JSON does not (seem)
to (safely) allow placing arbitrary bytes in strings; google for JSON
binary data and you'll find plenty of threads moaning the fact.

The bottom line is that YAML (and to a lesser extent, JSON) are meant to
be human-readable and portable data representations. Supporting binary
data in the way you suggest - tempting as it may be! - is simply not
compatible with these goals.

That said, you raise an interesting question about "almost UTF-8" data,
such as UNIX file names, value of tags in a TIFF file, etc. The best I
can offer (this is something I saw in some of the JSON threads) is that
in such cases, you use an _additional_ escaping mechanism; the obvious
choice is to use URL-like %xx encoding inside your strings. This would
allow for encoding arbitrary bytes. However, the YAML parser itself will
not be aware of this; the application will need to decode/encode the
data itself, possibly using a different tag (would !!str#url work?
Hmmm...).

> IS_BREAK is somewhat annoying. I think it may be a good idea to redefine 
> breaks as only being NL

Not going to happen. Too many windows text editors and other
applications generate CR-LF pairs, and once you support that there
is really no reason not to support Mac's CR-only files as well.

Finally, as for using/patching Syck... We really should put Syck back
into the grave; it isn't even YAML 1.1, never mind 1.2. It is hopelessly
obsolete. A better use of resources would be to create a Syck-compatible
wrapper for libyaml so we can use it as a drop-in replacement. We really
need some volunteer to do that...

Have fun,

    Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july