Re: Invalid UTF-8

Oren Ben-Kiki wrote:

> If I try to load them using a YAML
> library that uses UTF-16 (e.g., a Java one), and the \X escape sequence
> specifies an invalid UTF-8 byte sequence, then the library will be
> forced to throw an exception. So the file is not "portable" to such a
> system (cannot be loaded by it).

Okay it sounds like the only acceptable solution to you is to use %nn 
encoding, since the strings are then convertable to UTF-16. A few things 
I have learned about this:

1. Use %25 to quote %. This avoids exponential explosions of % signs 
when run through further layers that use %% for quoting. "\%" would be 
nice but is not possible with how yaml is returning the strings.

2. Libyaml also barfs on invalid UTF-16 so there is a need to store that 
as well. My recommendation is to store this as three %nn sequences per 
surrogate half showing the UTF-8 encoding of that invalid UTF-16. This 
will allow the same decoder to be used for both invalid UTF-8 and UTF-16.

I am not going to do this. For our software what I plan to do instead is 
use my patched version but modified to use lower-case \x instead of 
uppercase. This means the files will be readable on other yaml 
implementations, only invalid UTF-8 will get screwed up but it won't 
throw an error.

I do want to submit my patch for libyaml, with changes so that it 
restores throwing errors on invalid UTF-8 input. I fixed a number of 
bugs and simplified the parser a lot by treating UTF-8 as bytes. In 
particular it will not address off the ends of buffers. I'll try to get 
this asap.

> Fair enough. YAML really doesn't care about filesystems and the whole
> issue is secondary, so lets drop it. For those interested :-)
> http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html contains
> a nice long rant about UNIX file systems

Yes I have read that paper. I think it is pretty clear that he also 
wants to assume UTF-8 at all times, but also realizes that this cannot 
work if invalid sequences produce an error and cannot be processed. He 
proposes the same Python utf8b solution I did, which Python also calls 
PEP383, more on this below.

Most of his comments are about control characters, which is also why I 
don't like the fact that libyaml adds NEL and 2 other sequences as yet 
more control characters! I would like to see libyaml altered so on 
output it always quotes these as \uNNNN sequences.

Windows allows NEL and DEL and some other control characters, it allows 
invalid UTF-16, and lots of punctuation marks including quotes, and 
leading/trailing spaces (this is more common on Windows due to bugs in 
gui programs). His claim that these are somehow "Unix problems" is quite 
misleading. I think his final conclusions that low-level implementation 
must somehow reject a rather complicated pattern is impractical. He 
should instead be blaming the shells and other software that does not 
offer any practical way to quote an arbitrary byte stream, all his 
examples were of such programs.

> Now we are getting somewhere... so basically you are converting between
> invalid UTF-8 byte sequences to (some) invalid UTF-16 byte sequences in
> a lossless fashion; presumably it is possible to expand this to use
> invalid UTF-32 byte sequences as well.  Such a system could in theory be
> supported by YAML, because it would allow a library to manage to load
> the file regardless of the encoding used.

My personal belief is that the best solution is to use byte streams at 
all points. This is because invalid UTF-16 can be losslessly translated 
to UTF-8, and so can any byte encoding (by using the bytes unchanged). 
This makes UTF-8 a safe way to losslessly store all possible strings. It 
could also store invalid UTF-32 by extending UTF-8 encoding to the 
original 31 bit specification and add a way to store a 32-bit number.

This is not true of UTF-16 and I'm not sure if it can be.

PEP383's solution is turning invalid UTF-8 bytes into 0xDC80..0xDCFF and 
also (they don't mention this) declaring the UTF-8 encodings of all 
surrogate halves 0xD800..0xDFFF invalid (and thus they turn into 3 
0xDCxx words). Now UTF-16 can safely store any byte stream. But the 
translation from UTF-16 to UTF-8 is no longer lossless, and since we are 
going to want to write filenames to 8-bit text streams this is a serious 
problem. PEP383 basically punts and suggests techniques that are 
equivalent to tracking the original encoding, but if I could do that I 
could leave the data in it's original encoding!

Also the translation is seriously incompatible with the CESU encoding 
used by virtually all software, and there is the danger that a stupid 
decoder will turn a sequence of 0xDCxx bytes into a valid UTF-8 encoding.

A truly lossless bidirectional conversion between UTF-8 and UTF-16, 
where invalid strings map into invalid strings, would solve this (it 
would also handle UTF-32->UTF-16 by first going to UTF-8). But I am 
afraid it may be fiendishly complex or impossible:

The idea I have is to start with PEP383 with no surrogate halves and a 
really stupid UTF-16->UTF-8 converter that turns all mismatched 0xDCxx 
back into the raw byte.

Then find every legal UTF-8 string with surrogate halves in it that when 
translated to UTF-16 and back are unchanged, and declare these "valid". 
For instance all the low surrogate halves 0xDD00..0xDFFF can be valid. 
And the high surrogate halves can be valid if not followed by a low 
surrogate half or an error.

Then make the UTF-16 converter smarter, by examining any sequence of 
0xDCxx words. If these turn into a "valid" UTF-8 string then instead of 
converting them to bytes, convert the first one to the correct decoding 
of 0xDCxx.

This will increase the set of "valid" UTF-8, so change the encoder to do 
these unchanged. And change the decoder to not decode and 0xDCxx into 
these new valid sequences, which will increase the valid set even more. 
Repeat until all UTF-8 strings are "valid".

My worry is that either this results in an infinitely complicated set of 
rules, or you soon arrive at a point where you cannot make any more 
strings valid.

> I would not object _in principle_ for extending YAML in that direction,
> _given_ such a mechanism became somewhat mature/acceptable (e.g., a
> clear definition somewhere, adoption by some tools such as Java, iconv,
> Unicode processing libraries), etc. I'd be _very_ wary of blazing the
> trail here.

I agree with this. However I do want to make sure that people 
acknowledge the problem. Trying to "solve" it by defining the problem as 
non-existent is pointless and burying your heads in the sand.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july