Re: New invalid UTF-8 proposal using tags and %nn

On Thu, 2009-09-03 at 13:03 -0700, William Spitzak wrote:
> UTF-8 TAG PROPOSAL:

I wouldn't call it !!utf8, since technically it isn't. Calling it !!utfx
would be confusing as this is not compatible with Perl's UTF-X. How
about calling it !!utf-u (UTF + URL encoding, or
UTF-"Universal"/UTF-"Unlimited")?

> * A '%' sign must be written as "%25" if it would otherwise be misread 
> by the parser.
>
> * A writer *MAY* write any byte >= 0x80 and '%' as "%nn".
> 
> * The reader turns a '%' followed by "25" into a '%' sign, and '%' 
> followed by "80".."FF" hex into a raw byte. Otherwise the '%' is 
> literal, this includes '%' followed by hex less than 0x80.

So, say, %20 would not be a space? It would be '%' '2' '0'? IMO this
would be very confusing. I think it would be much better to say that '%'
must always be followed by 'nn' and is always interpreted, and '%25' is
always needed for encoding '%'. This would be safer, less surprising,
and also compatible with URL encoding.

> UTF-16:

The main question about UTF-16 here is: What should a Java library (or
any library that uses UTF-16 or UTF-32 natively) do when loading this
tag? In fact, what should a library using UTF-8 do? If I understand you
correctly, your proposal is that a library would:

1. First, load the scalar into a byte array (not a character array), not
a character sequence. "%nn" is loaded to a byte with the value "nn". The
byte array uses (invalid) UTF-8 encoding, regardless of the platform
used by the library (that is, even in a Java YAML library). This is
always safe and always possible. A library API may expose this byte
array directly.

2. If the platform used by the library happens to use UTF-8 (e.g. a C
YAML library), _and_ if it is accepts invalid UTF-8 byte sequences
inside its built-in UTF-8 string type (I suppose C qualifies here :-),
the library may return the byte array created in step 1 as if it was the
built-in string type (in C it is vague anyway).

3. If the platform used by the library uses some other encoding (say,
Java using UTF-16, or some hypothetical system using UTF-32), convert
the (invalid) UTF-8 byte array created in step 1 to an (invalid)
UTF-16/UTF-32 word array. In this process, perform two things
differently from the normal UTF-8 to UTF-16/UTF-32 conversion process:
(A) allow the creation of arbitrary words, such as 0xFFFF, a
free-floating 0xDCxx, etc.; and also (B) convert any free-floating byte
with the value 0x80 and above into the word 0xDCxx (for UTF-16), or
0x0000DCxx (for UTF-32). This is also always safe and always possible. A
library API may expose this word array directly.

4. If the platform used by the library used UTF-16 or UTF-32, and _if_
it accepts invalid UTF-16/UTF-32 words inside its built-in UTF-16/UTF-32
string type (I don't know if Java qualifies here, but it may), the
library may return the word array created in step 3 as if it was the
built-in string type.

Of course the library could collapse several steps together, e.g. load
the string directly into a UTF-16 Java string class, without an actual
intermediate UTF-8 byte array; the above are "conceptual" steps, not
necessarily "physical" ones.

There is one problem with the above; the process of loading into a
UTF-16/UTF-32 word array is not lossless. That is, there is no way to
distinguish between an input "%nn%nn%nn" that expands to "0xDCFF" and a
the input "%ff". Both would become the same UTF-16/UTF-32 word (0xDCFF).
So, the definition of this tag _must_ specify that the two forms are in
fact "the same" and that one of them (presumably the simple "%ff") is
the "canonical" form.

> ADVANTAGES:
> 
> * The primary advantage of this scheme is that unaware yaml processors 
> will not mangle the % quoting when copying the file.

Yes.

> * The valid UTF-8 and ASCII portion of the string is readable and 
> editable in a text editor. This encourages users to convert to and use 
> valid Unicode. Most other proposals have the opposite effect.

Yes.

> * Similar to the "binary" base64 proposal.

Yes.

> * Matches how yaml tags are written and how bytes are quoted in URLs.

Yes.

> * Uses 3 characters rather than the 4 used by "\XNN".

Yes.

Also:

* Is portable between platforms regardless of their preferred native
Unicode encoding.

* Allows using the built-in "string" type of the platform if possible.

> DISADVANTAGES:
> 
> * Adds a second escape character rather than re-using '\'.

Alas...

> * Tag is required, just like "binary". This means tag cannot be used for 
> it's original purpose, and the file cannot be converted to JSON.

Yes. This is also a problem with base64 (!!binary). Basically, in JSON,
there is an implicit assumption that the application consuming the data
somehow knows the right type (tag) of each field (presumably, based on
the path leading to it). So it isn't that you can't use this mechanism
in JSON, it is just that any application consuming it needs to know that
the value of the 'foo' field needs to be decoded (just like if it was
binary).

> * %-encoded URL's and printf formats with a width after the '%' are mangled.

Right. But on the bright side, there's no reason to ever use !!utf-u for
URLs. And passing printf format strings is hardly a common use case, so
suffering a bit there isn't a big deal, in the grand scheme of things.

> * Without modifying libyaml, requires a slow second pass to examine 
> strings and potentially allocation of another temporary buffer to hold 
> the converted string.

You must have a decoding pass (just like for !!binary). Given the
decoding (in a UTF-8 system such as libyaml) is always _shrinking_ the
size of the string, it is easy to do this "in place" inside the original
buffer, so no additional memory would be needed. So even if you do it
after libyaml has given you the "raw" scalar, I don't see there's a
significant performance penalty. Also I believe libyaml allows you to
register your own tag handlers, so this could be made transparent to the
application.

> FINAL COMMENTS:
> 
> Like the base64 proposal, this can be entirely done by the program 
> calling yaml. However it helps considerably if everybody can agree on 
> the tag and various nuances of the encoding, so there are not a hundred 
> variations.

Oh, for sure. This is why we have a tag repository.

> I do seem to be having difficulty conveying why this is necessary, but 
> you have to believe me that we will NEVER see Unicode used uniformly for 
> byte files unless this is supported.

I never argued against the need, just the specific solutions. In
particular, portability of the same YAML file between different systems,
regardless if these systems use UTF-8 or UTF-16 or UTF-32, is key. This
proposal seems to satisfy it (with the tweaks I gave above).

!!utf-u is useful for scalars that are _almost_ Unicode with the
occasional binary noise thrown in. As you point out, such cases happen
often in the real world (as opposed to a perfect world where everyone
used Unicode). URLs faced the same problem and essentially solved it in
the same way, so being compatible with them makes sense.

Have fun,

    Oren Ben-Kiki



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july