Re: Invalid UTF-8

Oren Ben-Kiki wrote:

> We are in luck. The Unicode standard is defined such that all the 8-bit
> values (0-255) are valid code points, so it is possible to use "\xNN" to
> represent any "byte" in a YAML string.

No, the sequence \xNN turns into a valid 2 byte sequence in UTF-8 and 
does not serve this purpose.

I think you are AGAIN proposing the "solution" of double-encoding the 
UTF-8. At first appearance you may think you have just made all 
non-ASCII unreadable, but if you examine the "unreadable" Unicode you 
will realize that you have just defined the encoding as ISO-8859-1. This 
is EXACTLY what I am trying to prevent!

As I said before, the vast majority of "solutions" are the same as 
saying "all Unix filenames are ISO-8859-1". You and several other 
posters here have now managed to suggest this about a dozen times, 
apparently without realizing what you are doing. (the only suggestion 
that is not equivalent to ISO-8859-1 only is the '%' encoding).

That said, I would favor a redefinition of yaml so that \xNN means the 
raw byte (to write what \xNN does currently for 0x80-0xFF, you must 
instead write \u00NN). The main reason for this is to be compatible with 
C strings. My current plan is to patch libyaml to do exactly this.

> However (1) *Not* all 16-bit
> "\uNNNN" or 32-bit "\UNNNNNNNN" are valid and (2) this doesn't solve
> William's problem.

I believe you are referring to the idea that the "surrogate halves" 
codes of 0xD800..0xDFFF are somehow invalid and should not be allowed.

This is entirely due to a defect in UTF-16's design. They should have 
defined a method of encoding those values, just like UTF-8 defines a 
method of encoding the values 0x80-0xFF.

I believe \uNNNN should always work even for surrogate halves. Yes a 
pair might turn into "valid" UTF-16. But this is also true of the UTF-8 
\xNN or %nn solutions. Even unmodified libyaml turns "\x41" into the 
same thing as 'A'. The fact that the same string can have multiple 
escaped representations is only an argument that escapes should be 
removed at a low level, such as in libyaml, this is why I do not like 
schemes that post-process such as the suggested '%' escaping.

At a practical level you have to support arbitrary \uNNNN because 
otherwise you cannot name Windows filenames. This is just as stupid as 
breaking Unix by disallowing invalid UTF-8 with the addition that even 
"double encoding" does not work!

(I suppose that if invalid UTF-8 is accepted you could require that 
surrogate pairs be quoted as 3 invalid UTF-8 bytes but I don't think 
people working with UTF-16 would consider the result readable)

UTF-16 has ENORMOUS problems and people are ignoring them because they 
are scared to death that they will be labelled "politically incorrect" 
by having the audacity to suggest that English get the "better" shorter 
encodings. This has led to insanity such as claiming the surrogate 
halves are invalid, because they cannot find any solution to their 
failings other than forcing UTF-8 to be just as difficult. But they have 
even failed at being "politically correct": attempts to force Unicode to 
fit into 64K led to HAN unification, which (measured by population) has 
pissed off far more people than making English "shorter" would have!


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july