Re: Invalid UTF-8

Oren Ben-Kiki wrote:

> I don't see where you get the idea I am promoting ISO-8859-1 encoding.
 > All I said was that you can use "\xNN" for any value of NN. Yes, some 
 > of these become two-byte sequences in UTF-8. Nothing more, nothing
 > less.

There is "something more": You missed the fact that the Unicode 
characters 0x80..0xFF must turn into something else, otherwise they 
cannot be distinguished from the raw bytes. (a possible solution that I 
assummed is "double encoding" where two Unicode characters are written 
that match the UTF-8 encoding). No matter how you solve this problem, 
the chracters 0x80..0xFF are mangled in the resulting file!

Now look what happens if the output converter is given an ISO-8859-1 
string. Look at the result: each byte (except for the extremely unlikely 
chance that they are a valid UTF-8 sequence) is turned into \xNN and 
thus into the matching ISO-8859-1 Unicode character!

And look what happens when a user edits the yaml file in a Unicode 
editor and pastes in some text. All characters in the range 0x80..0xFF 
will turn into raw bytes! If these are the majority or only non-ASCII 
then in fact the resulting name is in effect ISO-8859-1!

Please understand what users will see: text in ISO-8859-1 is READABLE, 
while UTF-8 is UNREADABLE! I don't care how much you say the encoding is 
"UTF-8", it is ISO-8859-1. You have basically defined all byte streams 
that contain ASCII letters as being ISO-8859-1! And the shameful part is 
that you sincerely believe that by you are encouraging Unicode, while 
actually you are PREVENTING it!

> You keep saying that but it makes no sense and I am completely baffled
> by it. For the record, and for the last one, I do not suggest YAML uses
> any encoding other than Unicode (UTF-*), under any circumstance, at any
> place, in any library, file, API, anywhere, *ever*.

This has NOTHING to do with the encoding of the yaml file itself. I 
fully support libyaml writing only valid UTF-8 and it probably is not a 
huge deal if only valid UTF-8 is accepted on input.

What I need is a lossless way to store invalid UTF-8 in a scalar, 
without making valid UTF-8 unreadable in the resulting file. The file 
format itself can be valid UTF-8 or UTF-16 or UTF-32. I'm sorry my 
initial posts confused this with the (perhaps unrelated) ability to read 
invalid UTF-8 files.

>> I believe you are referring to the idea that the "surrogate halves" 
>> codes of 0xD800..0xDFFF are somehow invalid and should not be allowed.
> 
> Not only that. "\uFFFF" is explicitly forbidden, for example.

Not in any documentation I have ever seen. Although I think it is 
misguided, they make a huge distinction between the surrogate halves 
0xD800..0xDFFF and the 36 "noncharacters": U+FDD0..U+FDEF and any code 
point ending in the value FFFE or FFFF. Noncharacters are explicitly 
allowed by both UTF-8 and UTF-16 encodings. You may want to check up on 
this.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july