Re: Invalid UTF-8

Oren Ben-Kiki wrote:

> what is the expected result if a Java YAML library attempts
> to read the same file?

It will throw an error, JUST LIKE IT DOES NOW! Boy, that was a hard!

> The spec makes such invalid bytes an error. This ensures portability.
> Any YAML library in any language using any encoding is guaranteed to
> load  the "same" data from the file. We do _not_ want to give this
> property up!

But EVERY ONE of your "solutions" is not portable beause a program 
reading the file will get a different string than I attempted to write, 
even for valid UTF-8 (such as a the 0x80-0xFF Unicode characters, or a 
'%' sequence, or binary). So "portability" is not an explanation for this.

> Now let's look at your use case of stuff (say, file names) that "should"
> be Unicode but actually is not. In which case you have several options.

> One, sanitize the data so it is valid Unicode. Treat it as a data entry
> problem. This is obviously the best option (once applied there's no more
> problems) and is also the worst one (since it may not be possible).

My patch wrote "\XNN" to the file. This is four valid Unicode 
characters. Therefore it sure did "sanitize" it! I did exactly what you 
suggested.

I think you are actually suggesting that "%nn" or some similar escape 
that does not use backslash be written so that libyaml does not have to 
be modified. This is a workable solution. I just don't like it because I 
don't like making two different escape characters with different methods 
of quoting them, and would greatly prefer a one-pass algorithm.

> Two, if the data uses some other encoding (say, ISO-whatever), it is by
> definition using a subset of Unicode. You can trivially represent it as
> a Unicode string (using UTF-8 or whatever) using the proper character
> mapping tables. If the application reading the file wants to emit the
> data in ISO-whatever encoding, or in Morse code, or whatever, it is
> welcome to do so using similar character mapping tables.

Okay, here you *literally* said "Unix filenames are not UTF-8". Example #1.

> Three, accept the fact the data is _not_ Unicode. Treat it as a binary
> blob. This means that when writing the YAML file you must convert this
> binary data to a string (a sequence of valid Unicode code points); the
> applications will similarly need to convert this string back to the
> original binary blob.

Here you are suggesting that all filenames be written as binary blobs.

> There are many ways you can convert binary data to a Unicode string. One
> is to use base64 (YAML calls this !!binary). This is suitable for "true"
> binary data.

Here you suggest that the binary blob only be written if the UTF-8 is 
invalid. This uses up the tag so it can't be used for any other purpose. 
Also if I'm going to use the tag, I might as well use %nn syntax, which 
is more readable and leaves ASCII and valid UTF-8 readable.

> Another way is to emit each byte as a separate Unicode character, and
> use "\xNN" escape sequences for bytes whose value maps to a
> non-printable character (YAML currently has no standard name for this,
> but of course allows it). This is suitable for "almost-ASCII" data such
> as file names.

Here you suggest writing all UTF-8 filenames as double-encoded UTF-8, 
this is a common "solution" which is equivalent to "Unix filenames are 
ISO-8859-1 only".

> Again, if we had allowed you to specify "invalid UTF-8 bytes" inside the
> string (say by using "\Xnn" as you proposed), and the YAML library
> happened to be written in Java, what do you expect it to do? There's
> simply no right possible answer.

Yes there is a right answer: it throws an error, JUST LIKE IT DOES RIGHT 
NOW.

Maybe this would be more obvious if you imagine adding a new api to the 
C version that returns UTF-16 and throws an error, and a new api to the 
Java version that returns an array of bytes and does not throw an error.

>> Or look at your handling of UTF-16. Are mismatched surrogate halves a 
>> "data error"? ...
> 
> This is a great example. If I have a library that _always_ returns UTF-8
> and I see an input with mismatched surrogate halves... what should I do?
 > (iconv...)

I specifically asked for a UTF-16 handling program that threw errors on 
mismatched surrogates. You instead gave me a program that reads UTF-8!

I want to see a program that throws an error when given UTF-16 with 
invalid surrogates halves in it. How about a windows filesystem api that 
disallows mismatched surrogates. Or a Python implementation where you 
cannot strcat two UTF-16 strings if they have mismatched surrogates. 
Something like that.

And iconv (at least mine though this is a rather old Linux) allows 
invalid surrogate pairs when converting UTF-16 to UTF-16. So it is in 
fact a perfect example of this hypocrisy.

> Actually, I was referring to the fact you can encode _any_ 8-bit *binary
> blob* byte using "\xnn" notation, that is the second way to encode
> binary data I described above.

Once again you repeat the "Unix filenames must be ISO-8859-1 only" 
statement.

This isn't an empty statement because it
> is _not_ the case you can encode any 32-bit binary blob word using
> "\unnnn" notation (e.g., "\uFFFF" is _not_ valid). All this has
> _nothing_ to do with encodings.

\uFFFF is valid Unicode and libyaml accepts it! Are you talking about 
surrogate halves? They are \uD800 through \uDFFF.

And UTF-8 could encode 31 bit numbers originally (and 32 but there were 
two different schemes). The restriction is artificial and due to UTF-16!

> Obviously if you knew what encoding was used by the file name in
> question you could trivially convert it to Unicode and there wouldn't be
> any problem in the first place, as discussed above.

Yet again you say "Unix filenames are restricted to ISO-8859-1"

> Actually, Plan9 _invented_ UTF-8. All their file names are valid UTF-8 -
> I'm almost certain there's no way you can create a Plan9 file name with
> invalid UTF-8 bytes in it, which means there's no problem just pasting
> them as-is into a YAML file. Not that this matters.

Plan9 allows filenames to be arbitrary byte sequences, except they 
cannot contain nul or slash in them. 9FS does not have any idea of 
encoding what so ever.

>> Unix can handle Unicode and the filenames are UTF-8.
> 
> Great. If you know they are UTF-8 just paste them into the YAML file
> as-is.

I "know thay are UTF-8" is because THEY DO NOT HAVE INVALID UTF-8 IN 
THEM! But if it is impossible to store the counter examples, then I 
cannot do this! You are again forcing Unix filenames to be ISO-8859-1 only.

> Besides, it isn't just me;
> let's see how far you'll get convincing the iconv maintainers they
> should pass such invalid bytes untouched if iconv happens to convert
> from UTF-8 to UTF-8 :-)

Yea right I'm convinced by Ulrich Drepper, the person who rejected strlcpy.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july