Re: Invalid UTF-8

On Fri, 2009-08-21 at 12:44 -0700, William Spitzak wrote:
> Your outlook is polluted by thinking the only thing you can do with 
> "Unicode" is convert it to UTF-16 immediately. The correct solution, 
> which is hard for people used to UTF-16 to see, is to LEAVE IT AS UTF-8! 
> Try this before you say anything stupid. Or just take a look at the 
> libyaml source code, which works this way.

Ok, we clearly have a confusion with regard to the role of encodings
here.

The YAML spec clearly defines strings as a sequence of Unicode code
points. This has nothing to do with how these are encoded in the YAML
file or the in-memory native data object. Of course the input file may
use \x \u or \U escape sequences - but still, these must all expand to
valid Unicode code points.

As far as the Unicode standard is concerned (and therefore the YAML
spec), nothing other than Unicode code points exists. Everything
imaginable is expected to be expressible in terms of these code points
and only these code points. Period.

Nothing above is tied to a specific encoding. All encodings are treated
equally - UTF-8, UTF-16, and UTF-32 are all on equal footings here. Even
UTF-32 is a "specific" encoding since it specifies the byte order
(UTF-32LE or UTF-32BE). Certainly (the two variants of) UTF-16 are not
special in any way.

Also, the input file may use different encoding than the
library/application. For example, the same file will be loaded to UTF-16
in-memory String objects in Java but to UTF-8 string objects in Python,
regardless of whether the input file used UTF-8, UTF-16 or UTF-32
encoding.

It follows that every library needs to be able to convert from
UTF-whatever-the-input-was to the
UTF-whatever-the-programming-environment-requires. Sometimes both
encodings are the same, in which case the library can simply pass the
bytes along. Sometimes this isn't the case, and the library needs to
perform a conversion. It always needs to expand escape sequences, of
course.

So, it may _seem_ a "reasonable" hack to allow "invalid bytes" to be
passed on by the library if the two encodings are the same. However,
there's no sensible way to define what the library should do to "invalid
bytes" if the encodings are _not_ the same. That is, perhaps your
patched libyaml will happily pass invalid UTF-8 bytes to your UTF-8 C
program, but what is the expected result if a Java YAML library attempts
to read the same file?

The spec makes such invalid bytes an error. This ensures portability.
Any YAML library in any language using any encoding is guaranteed to
load  the "same" data from the file. We do _not_ want to give this
property up!

Now let's look at your use case of stuff (say, file names) that "should"
be Unicode but actually is not. In which case you have several options.

One, sanitize the data so it is valid Unicode. Treat it as a data entry
problem. This is obviously the best option (once applied there's no more
problems) and is also the worst one (since it may not be possible).

Two, if the data uses some other encoding (say, ISO-whatever), it is by
definition using a subset of Unicode. You can trivially represent it as
a Unicode string (using UTF-8 or whatever) using the proper character
mapping tables. If the application reading the file wants to emit the
data in ISO-whatever encoding, or in Morse code, or whatever, it is
welcome to do so using similar character mapping tables.

Three, accept the fact the data is _not_ Unicode. Treat it as a binary
blob. This means that when writing the YAML file you must convert this
binary data to a string (a sequence of valid Unicode code points); the
applications will similarly need to convert this string back to the
original binary blob.

There are many ways you can convert binary data to a Unicode string. One
is to use base64 (YAML calls this !!binary). This is suitable for "true"
binary data.

Another way is to emit each byte as a separate Unicode character, and
use "\xNN" escape sequences for bytes whose value maps to a
non-printable character (YAML currently has no standard name for this,
but of course allows it). This is suitable for "almost-ASCII" data such
as file names.

In _both_ cases you need a reverse conversion step to be applied to get
the original binary blob. Note this second reverse conversion step is
required _regardless_ of the encoding used by the input file and the
programming environment. Even if it is UTF-8 all the way, you _still_
need to _undo_ the conversion that was done when the file was written,
because the YAML file contains a _string_ and what you want is the
original _blob_.

There is _no_ way you can get the YAML library to magically use its
Unicode processing methods to _directly_ produce the binary blob you
want, since it is _not Unicode data_. You must have a conversion from
whatever-mess-it-was to Unicode and the reverse conversion from Unicode
to the original-mess-it-was.

Again, if we had allowed you to specify "invalid UTF-8 bytes" inside the
string (say by using "\Xnn" as you proposed), and the YAML library
happened to be written in Java, what do you expect it to do? There's
simply no right possible answer.

And please, let's not get personal here:

> Or look at your handling of UTF-16. Are mismatched surrogate halves a 
> "data error"? ...

This is a great example. If I have a library that _always_ returns UTF-8
and I see an input with mismatched surrogate halves... what should I do?
UTF-8 does not support surrogate pairs (see
http://unicode.org/faq/utf_bom.html#utf8-5 for example), so it _must_ be
an error. And before you dismiss this as "academic", the iconv program
_used_ to allow for UTF-8 encodings of surrogates pairs but this was
treated as a bug and was fixed:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=525299 . The iconv
program is as real-world as it gets.

> ... I want to see you fix your code, since you claim it is less
> "agony". Oh it is a bit difficult, huh? Your code got much longer and 
> slower? 

Libyaml was written by Xitology. That said, I am the one who maintains
the YAML _specifications_ (together with Clark and Ingy), which drive
the implementations, such as libyaml. Libyaml is behaving as per the
spec, so your beef is with me (and Clark and Ingy), not with Xitology.

"My code" is the YAML Reference implementation (written in Haskell). It
flags such pairs as an error, and has done so from day one. This
actually _saved_ me lines of code and has no measurable effect on
performance. My library is dog-slow, of course, but that's because of
completely unrelated issues. I don't think this issue has any measurable
impact on the performance of any library's Unicode processing code.

> > BTW - Unicode does allow for _all_ 8-bit characters. You can argue about
> > their semantics, but the fact is that a simple "\xNN" escape sequence
> > inside double-quoted strings _will work_ for all the 256 single-byte
> > values. So, for the case of stuff like UNIX file names, I really don't
> > see the problem.
> 
> You have just said "Unix filenames are ISO-8859-1 only". Yea, it's 
> really old fashioned and only used by fat nerds in their mothers 
> basement, too.

Actually, I was referring to the fact you can encode _any_ 8-bit *binary
blob* byte using "\xnn" notation, that is the second way to encode
binary data I described above. This isn't an empty statement because it
is _not_ the case you can encode any 32-bit binary blob word using
"\unnnn" notation (e.g., "\uFFFF" is _not_ valid). All this has
_nothing_ to do with encodings.

Obviously if you knew what encoding was used by the file name in
question you could trivially convert it to Unicode and there wouldn't be
any problem in the first place, as discussed above.

> I think the users of Linux and Samba and Plan9 (from 20 YEARS AGO!!!) 
> and Inferno and OS/X (which has the best Unicode support in the world) 
> may disagree with you a bit.

Actually, Plan9 _invented_ UTF-8. All their file names are valid UTF-8 -
I'm almost certain there's no way you can create a Plan9 file name with
invalid UTF-8 bytes in it, which means there's no problem just pasting
them as-is into a YAML file. Not that this matters.

> Unix can handle Unicode and the filenames are UTF-8.

Great. If you know they are UTF-8 just paste them into the YAML file
as-is. Of course, "UNIX" is a whole set of systems (Solaris? Bsd?
Linux?) and whole problem is when file names are _not_ UTF-8 and not
UTF-16 and not UTF-32 and not known to be ISO-whatever - in general,
when they are not *Unicode* and you have no idea how to *convert* them
to Unicode.

> > Either way, I simply don't see the use case for supporting invalid
> > Unicode characters, either as raw bytes or as escaped characters.
> 
> All I can say is that I suggest you try writing some software using byte 
> arrays and get back to me then.

Well, I have written quite a bit of that, thank you very much. I
happened to write a full commercial text indexing and retrieval engine
in the early 90s. I had to deal with a zillion ways to encode text, some
fixed-width, some variable-width, having to _guess_ which encoding to
use based on heuristics from hell, and so on - and that's on top of the
joys of having to deal with the file formats. Besides, it isn't just me;
let's see how far you'll get convincing the iconv maintainers they
should pass such invalid bytes untouched if iconv happens to convert
from UTF-8 to UTF-8 :-)

Have fun,

   Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july