Re: New invalid UTF-8 proposal using tags and %nn

Oren Ben-Kiki wrote:
> On Thu, 2009-09-03 at 13:03 -0700, William Spitzak wrote:
>> UTF-8 TAG PROPOSAL:
> 
> I wouldn't call it !!utf8, since technically it isn't. Calling it !!utfx
> would be confusing as this is not compatible with Perl's UTF-X. How
> about calling it !!utf-u (UTF + URL encoding, or
> UTF-"Universal"/UTF-"Unlimited")?

I did not like using a '-' because first of all I did not know it was 
valid in a tag, but also because "binary" does not have it, and it 
prevents the tag from converting to a class identifier in most 
programming languages.

I do not want a tag that implies the sting is "invalid", because you can 
write valid strings using it, and it is likely programs that don't want 
to test the strings will do exactly that.

"url" implies that all bytes with the high bit set, and many others, are 
encoded with %. This is allowed but not emphatically not required! So I 
am worried that "url" would be misleading.

I would prefer a short tag due to the fact that this may be produced in 
enormous numbers by programs, for instance they may tag *every* scalar 
with them.

This can be used to store invalid UTF-16, and even invalid UTF-32 
(provided you extend UTF-8 to cover 32 bits). So there is an argument 
that it should not imply UTF-8. However it certainly is using details of 
UTF-8 in the definition, and also I can imagine somebody wanting a new 
scheme for invalid UTF-16 (such as %NNNN) and wanting a different tag 
for that with "16" in it.

Therefore I still like "utf8" or "utf-8" the best. But "utf-u" is ok. 
Also "utf" and "url" because they are short.

One crazy idea is to call it "!!%25" encoding (which will look like "%" 
to the program calling libyaml).

> So, say, %20 would not be a space?

You changed my mind. I think %20 would be a space. As you point out it 
will not confuse URLs or printfs, both of which are likely to be written 
*without* this tag.

This would allow a non-quoted version of this, where %nn is used for 
every invalid byte and also for ASCII characters that have meaning to 
yaml. We do want to avoid making this a tempting method of dumping 
binary data, but it still expands slightly more than 2x, while base64 is 
a 4/3 expansion.

> I think it would be much better to say that '%'
> must always be followed by 'nn' and is always interpreted, and '%25' is
> always needed for encoding '%'. This would be safer, less surprising,
> and also compatible with URL encoding.

I consider it a requirement that "%%" be two '%' characters in a row. 
Otherwise we will get an exponential explosion when combining this with 
other systems that use "%%" to quote a '%'.

I also don't like throwing an error when there is no ambiguity in the 
expected result. And I want to disallow the temptation to add more % 
escapes (such as for invalid UTF-16). A new tag should be used for that.

> The main question about UTF-16 here is: What should a Java library (or
> any library that uses UTF-16 or UTF-32 natively) do when loading this
> tag?

The caller *must* be able to detect lossy conversion and I would prefer 
a design where it is hard for callers to ignore the error. Also I wish 
to make it easy to implement the reader by calling a language's built-in 
UTF-8->UTF-16 converter. Therefore what should be done:

1. The library *must* throw some kind of catchable error if conversion 
to UTF-16 and back to UTF-8 would be lossy. This means it throws an 
error on any invalid UTF-8 byte sequence, and also on the encoding of a 
high surrogate followed by a low surrogate.

2. The library *may* throw the same error on other strings, for instance 
on all surrogate halves.

3. The library *must* provide a way for the calling program to access 
the original bytes if such an error happens.

4. The library *may* provide a lossy conversion if such an error 
happens. In this case it may be a good idea to exactly define this lossy 
conversion. I strongly recommend surrogate halves be copied unchanged 
for compatibility with CESE/UCS-2. Error bytes should be converted 
individually, possibilities are:
   1. 0xDCxx (probably best)
   2. 0xFFFD
   3. The character from CP1252
   4. "%nn"

> In fact, what should a library using UTF-8 do?

I don't think there is any such library that does not accept invalid 
UTF-8. If there were then the same rules as for UTF-16 apply, it can 
throw an error but there must be a way to get the original bytes (it can 
ISO-8859-1->UTF-8 encode them if necessary).

> 4. If the platform used by the library used UTF-16 or UTF-32, and _if_
> it accepts invalid UTF-16/UTF-32 words inside its built-in UTF-16/UTF-32
> string type (I don't know if Java qualifies here, but it may), the
> library may return the word array created in step 3 as if it was the
> built-in string type.

Java and Python both accept invalid UTF-16 in their strings. They are 
well aware that nobody could use their software on Windows without this. 
Of course complaints from them about "invalid UTF-8" are just hypocrisy!

> There is no way to
> distinguish between an input "%nn%nn%nn" that expands to "0xDCFF" and a
> the input "%ff". Both would become the same UTF-16/UTF-32 word (0xDCFF).
> So, the definition of this tag _must_ specify that the two forms are in
> fact "the same" and that one of them (presumably the simple "%ff") is
> the "canonical" form.

NO! Those are two different byte arrays. Defining them as being equal 
defeats the whole purpose of this!

I recommend an output converter of invalid UTF-16 write 0xDCxx (and all 
surrogate halves) as the %nn%nn%nn sequence. This is for CESU/UCS-2 
compatibility.

>> * Without modifying libyaml, requires a slow second pass to examine 
>> strings and potentially allocation of another temporary buffer to hold 
>> the converted string.
> 
> You must have a decoding pass (just like for !!binary). Given the
> decoding (in a UTF-8 system such as libyaml) is always _shrinking_ the
> size of the string, it is easy to do this "in place" inside the original
> buffer

My concern was for *output*, not input.

> !!utf-u is useful for scalars that are _almost_ Unicode with the
> occasional binary noise thrown in. As you point out, such cases happen
> often in the real world (as opposed to a perfect world where everyone
> used Unicode). URLs faced the same problem and essentially solved it in
> the same way, so being compatible with them makes sense.

Yes this is a good explanation.

I do want to point out that the normal result of encountering such 
strings is for users to treat them as a non-Unicode encoding, because 
"make it not work" is considered 100 times more benificial than "keep 
all the foreign letters". Thus not being able to handle binary noise in 
UTF is harming the adoption of Unicode, despite the initial naive 
impression that it is helping.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july