Re: Invalid UTF-8

Oren Ben-Kiki wrote:
> On Tue, 2009-08-18 at 19:31 -0700, William Spitzak wrote:
>> This requires that we be able to losslessly store invalid UTF-8.
> 
> I'm pretty uncomfortable with this. Ideally, text strings should be used
> for text strings - which means valid Unicode characters. Binary data can
> use something like !!binary and base64 encoding.

Unfortunately in the real world invalid UTF-8 shows up. Denial of 
Service and/or lossy translation is not a "solution". It is a problem 
and thus a bug in libyaml, no matter how "uncomfortable" fixing it is.

 > won't ... be able to print it.

Right in the middle of your text, you actually realize the correct 
solution! Invalid UTF-8 is a DISPLAY/PRINT problem. It is not a data 
representation problem! It is the same as misspelled words or invalid 
sequences of Unicode combining marks. Deferring to display means that 
solutions that are unsafe or impossible (such as lossy decoding as 
CP1252, or displaying non-Unicode) can be used.

> There are two possible separate options for  support for invalid
> Unicode.  [ raw invalid UTF-8 and escape sequneces ]

The code I submitted always outputs \XNN sequences, which are four ascii 
letters and thus the files are always valid UTF-8. Thus concerns about 
text editors/etc are not relevant. This is also a requirement if you 
still want to write UTF-16 YAML files.

Reading invalid UTF-8 files is not a requirement for our uses (I expect 
hand-edited input to be ASCII-only), so I suppose a changed version that 
only does the \XNN escapes would be acceptable. But please see the 
bottom of this email for my rant on why I think treating invalid UTF-8 
input as a error is wrong and even evil.

> It
> means that trying to load such files into a Unicode-aware programming
> languages may fail.

Libyaml currently throws an error and refuses to read the file! So you 
are saying that "absolutely guaranteed to fail" is somehow worse than 
"may fail"? All software I have encountered either never translates the 
UTF-8, or calls a function such as Python decode() that throws an error 
anyway. And when that error is thrown I have the string from the file so 
I can maybe fix it!

I also do not like your definition of "Unicode aware" as "translates to 
UTF-16". It is true that Windows programmers seem to use this but it is 
wrong to say that. Programs using UTF-8 internally such as OSX seem to 
be *more* "Unicode aware" and far more likely to handle combining 
characters, canonicalization, sorting, and other aspects of Unicode, as 
opposed to programs that treat it as some kind of "big ASCII", and it is 
downright insulting to say this better software not "Unicode aware".

UTF-16's inability to losslessly represent an arbitrary byte sequence is 
a major problem but copying that bug to 8-bit files not not helping 
anybody! (I think the solution for UTF-16 is by translating invalid 
UTF-8 bytes to 0xCDxx but it is still lossy and it gets complicated...)

> FWIW, even JSON does not (seem)
> to (safely) allow placing arbitrary bytes in strings; google for JSON
> binary data and you'll find plenty of threads moaning the fact.

I am well aware that JSON has the same bug. It also does not handle 
UTF-8 encodings of anything after U+FFFF, but YAML does. So this is no 
excuse.

> That said, you raise an interesting question about "almost UTF-8" data,
> such as UNIX file names, value of tags in a TIFF file, etc.

"interesting" is the understatement of the year! Don't forget URLs.

> use URL-like %xx encoding inside your strings.

Having actually done exactly this I can tell you exactly where there are 
problems:

1. Running more than one escape-sequence remover introduces bugs. 
Running any more than once, or running in the wrong order, can make it 
very hard to track down misbehavior. Running them on strings that may 
have \0 in them can produce annoying bugs. Possible logical escapes such 
as "\%" are impossible.

2. Software is already using '%' as an escape so this reuse makes some 
input unreadable. In particular "%%" had better pass through unchanged. 
We required 2 hex digits for this to be recognized and the way to quote 
% is to put "%25" in. Still since the rules were not consistent with 
other software we had problems. Writers attempting to use these rules to 
minimize the need to quote % had bugs as this is not as easy as it 
looked (same reason YAML does not do \ooo octal sequences, btw).

3. Even with the final version, %-encoded URLs and printf formats with 
widths more than 9 were unreadable and users would type them in by 
accident and get invalid UTF-8, the very thing you think you are 
preventing with this misguided plan.

4. libyaml still has to be modified to accept invalid UTF-8 in the 
%-encoded tags. Since I have to modify it anyway, I might as well modify 
it to do things in a much safer and useful way where the escape starts 
with the same character as all other escapes.

>> IS_BREAK is somewhat annoying. I think it may be a good idea to redefine 
>> breaks as only being NL
> 
> Not going to happen. Too many windows text editors and other
> applications generate CR-LF pairs, and once you support that there
> is really no reason not to support Mac's CR-only files as well.

I didn't mean to exclude those. I meant to exclude the 3 "newline" 
characters that are > 0x80 in Unicode. I suspect these cause far more 
problems in text editors than invalid UTF-8 does! For compatability I 
would just change it to write these as \uNNNN in quoted strings, the 
parser can continue to special-case them as before.

> Finally, as for using/patching Syck... We really should put Syck back
> into the grave; it isn't even YAML 1.1, never mind 1.2.

Not sure what you are talking about here, you may be resonding to a 
different post?

NOTES ON HANDLING INVALID UTF-8 AS INPUT:

Throwing errors on invalid UTF-8 I consider very bad behavior and in 
fact is so damaging to I18N that I consider it to be at least culturally 
insensitive, and perhaps more evil that an outright racist programmer, 
due to the fact that it is more effective at excluding foreign languages:

For a programmer an error that makes a file unreadable is a SHOW-STOPPER 
BUG. However "losing accented letters" is a minor error (I have found 
this to be true even for French and German programmers).

Therefore when a programmer hits such an error, they will gladly and 
quickly replace the show-stopper with ANYTHING that fixes it and the 
literally DO NOT CARE what this does to internationalization. Based on 
dozens of occurrences, mostly with programmers writing C++ code to 
interface with Python, these are the solutions applied when they get an 
encoding error, starting with the most popular:

  1. Tell Python to use ISO-8859-1 even if you know the text is UTF-8

  2. Replace every byte with the high bit set with ASCII such as '?' or 
'\xNN', making the program ASCII-only.

  3. "translate" to UTF-16 by alternating the bytes with nul.

In no instance whatsoever have I seen a programmer try to "fix" the 
UTF-8, or test for correctness before doing the above translations, or 
decide to report the error to the user and refuse to read the file. So 
in fact the result is that the former Unicode program has been changed 
to ISO-8859-1 or even ASCII only.

But this is not the programmer being insensitive to I18N. They are 
simply not thinking about it. The insensitive person is the one writing 
the code that threw the error, since they certainly were thinking about 
it. They may feel they are forcing people to use Unicode correctly, but 
the result is the exact opposite, they are forcing people to restrict 
their programs to ISO-8859-1 or only ASCII. The result is 100% 
counterproductive, which is infuriating to me, and should be for everybody.

Bill Spitzak
Rhythm & Hues Software

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july