Re: Invalid UTF-8

Oren Ben-Kiki wrote:
> On Thu, 2009-08-20 at 18:44 -0700, Adrian Klaver wrote:
>> It is a data issue, pushing it up the stack only prolongs the agony.

This indicates that you have never tried to write any code to manipulate 
Unicode as byte arrays. It is about 10 million times EASIER to defer the 
errors until display.

Your outlook is polluted by thinking the only thing you can do with 
"Unicode" is convert it to UTF-16 immediately. The correct solution, 
which is hard for people used to UTF-16 to see, is to LEAVE IT AS UTF-8! 
Try this before you say anything stupid. Or just take a look at the 
libyaml source code, which works this way.

Or look at your handling of UTF-16. Are mismatched surrogate halves a 
"data error"? Or do you just defer it until display? Don't be a 
hypocrite, I want to see you fix your code, since you claim it is less 
"agony". Oh it is a bit difficult, huh? Your code got much longer and 
slower? The errors are thrown at inconvenient times? You don't seem to 
be able to remove the tests from the display anyway? You can't name some 
Windows files? Hmm...

And I completely fail to see see why having line 1 in this code sample 
throw an exception is better than line 2:

	1. yaml_parser_parse(&parser, &event);
	2. UTF16Encode(event.data.scalar.value);

If moving that error detection by ONE line is "agony" I'd hate to see 
your reaction to real programming problems...

> BTW - Unicode does allow for _all_ 8-bit characters. You can argue about
> their semantics, but the fact is that a simple "\xNN" escape sequence
> inside double-quoted strings _will work_ for all the 256 single-byte
> values. So, for the case of stuff like UNIX file names, I really don't
> see the problem.

You have just said "Unix filenames are ISO-8859-1 only". Yea, it's 
really old fashioned and only used by fat nerds in their mothers 
basement, too.

I think the users of Linux and Samba and Plan9 (from 20 YEARS AGO!!!) 
and Inferno and OS/X (which has the best Unicode support in the world) 
may disagree with you a bit.

Unix can handle Unicode and the filenames are UTF-8.

> You can even use this to encode arbitrary binary data, as long as you
> accept that (1) you are using a 4 YAML stream bytes ('\' 'x' N N) for
> each 8 bits of payload (except for say 1/4 of the bytes - call it 3
> bytes on average for random binary data), (2) that there will be an
> interim in-memory string representation using one "character worth" of
> bytes for each 8-bit payload (2 bytes if using UTF-16 like Java does,
> 1-2 bytes if using UTF-8 - call it 1.5 bytes on average for random
> binary data), and (3) that you'll need to execute string2binary on the
> interim in-memory string to convert it to a true binary bytes array.

(1) is incorrect. \xNN turns into TWO bytes (for NN>=0x80) so you cannot 
get arbitrary UTF-8. My original plan was to change libyaml to do 
exactly this, but that would be incompatible with existing yaml files, 
so I made the new \X escape instead (I also considered \nnn octal 
escapes but that was also incompatible).

(2) here you suggest that all UTF-8 be double-encoded. This makes 
non-ASCII unreadable which I don't think is one of YAML's design goals. 
If you only want to do this for invalid UTF-8 then you have to use a 
tag, in which case there are better schemes such as the %nn encoding, 
which the valid UTF-8 readable. I don't like using the tag as I want to 
use that for data interpretation, not file encoding.

> You'd be better off using !!binary and base64 for true binary data, of
> course - only 1.25 YAML stream bytes for each 8 bits of payload, and it
> will be loaded directly to an in-memory bytes buffer with no interim
> representation. But still, using "\xNN" for binary data may be a valid
> trick to use in some circumstances.

Here I believe you are suggesting that all invalid UTF-8 be written as 
!!binary with base64. The problem with this is that it is unreadable, 
even trashing ASCII letters! And, again, I want to use the tag to 
indicate data interpretation, not file encoding.

I suspect there is a worry that invalid UTF-8 will be used as a "binary 
compression scheme" and this is the real unstated objection to reading 
it. However a trivial way to prevent this would have been to disallow 
NUL characters (the fact that you don't indicates that somebody wants to 
dump raw UTF-16 without base64, which is just as bad!) Another solution 
may be to convert invalid UTF-8 bytes to the UTF-8 encoding of 0xCDxx 
characters, this means that invalid UTF-8 files will read but you cannot 
get arbitrary binary that way, the only problem is that it is not obvious.

> Either way, I simply don't see the use case for supporting invalid
> Unicode characters, either as raw bytes or as escaped characters.

All I can say is that I suggest you try writing some software using byte 
arrays and get back to me then.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july