Re: Invalid UTF-8

On Tue, 2009-08-25 at 18:26 -0700, William Spitzak wrote:
> Oren Ben-Kiki wrote:
> >> Putting Unix into a non-Unicode ghetto is not an answer.
> > 
> > Right - dragging UNIX (kicking and screaming) into the Unicode world
> > is :-)
>
> For some reason you believe you are helping Unicode with this behavior,

I have no special interest in "helping Unicode". All I want is the
security of knowing that a YAML file is portable across systems,
including C (UTF-8) and Java (UTF-16). If forced to choose between
having problems handling file names with weird bytes and giving up this
cross-platform safety, the answer (for me) is clear. The cross-platform
portability wins.

> but I am trying to clearly explain that this is by far the biggest 
> problem, much more so than completely ignorant programmers who think the 
> data is ISO-8859-1. At least their programs copy a UTF-8 file unchanged 
> and reliably.

See, this is the real problem. What encoding is used for file names? For
example, when you type "ls" in a shell inside an xterm window, and it
displays the file names, somewhere along the chain there _must_ be an
assumption about the encoding used for the file names. The POSIX
standard solves this by mandating that filenames must be encoded
according to the LC_CTYPE locale setting (I don't recall chapter and
verse off the top of my head; it has been a while).

What happens if the bytes in the filename are not valid according to the
LC_CTYPE locale setting? Good question... For example, what would a Java
program (that can only deal with UTF-16 characters) see when it fetches
the name of such a file? I'm not really certain but I suspect it isn't
pretty...

At any rate, YAML doesn't attempt to act as a "transparent pipeline"
allowing you to put anything-at-all on one side and get
exactly-the-same-thing on the other. It definitely _does_ have
restrictions about "what you can put in" (e.g., no duplicate keys in
mappings, so PHP must therefore use sequence-of-single-pair-mappings for
its weird data).

The burden is on whoever is pasting the filename into the YAML file to
figure out which encoding was used for that filename, and if he guessed
wrong, it is his responsibility to figure out how to deal with the '?'
or black square or whatever characters. BTW, if we are talking about
users doing copy-and-paste between an xterm and an editor window, than
this is easily visible and easy for the user to fix.

It is an explicit design decision to place the burden on the (always
single) author of the YAML file instead of the (possibly multiple)
readers of the YAML file. We insist that if the file is readable on any
YAML system, it will be readable on all YAML systems.

These design decisions, like all design decisions, involve a trade-off;
I can understand how your use case of "mostly-ASCII data with a few
8-bit bytes that maybe use some encoding I that don't know about" does
suffer as a result. Sorry about that... but we are not going to reverse
these decisions as they are core to the YAML concept.

> The person who is "kicking and screaming" is you, Unix has 
> correctly separated data and presentation for years.

Actually... as I pointed out, the POSIX spec _explicitly_ addresses this
as issue. IMO, if the LC_CTYPE locale is UTF-8, the kernel would be
within its rights to _reject_ any attempts to create files containing
invalid UTF-8 bytes, or at minimum convert them into an "equivalent"
valid UTF-8 string.

AFAIK Apple's OS-X does exactly that, since it uses UTF-16 internally in
HFS (even if it presents filenames as UTF-8 to the applications). Linux
does not do this; I'm not certain about BSD and Solaris and AIX and QNX
and all the other UNIXes out there. Then there's Plan9 ("UNIX as it
should have been") who _invented_ UTF-8 and I'm willing to bet that they
definitely enforce valid UTF-8 filenames. So the picture is not as
clear-cut as you make it to be.

Have fun,

    Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july