Re: New invalid UTF-8 proposal using tags and %nn

On Fri, 2009-09-04 at 12:52 -0700, William Spitzak wrote:
> I do not want a tag that implies the sting is "invalid", because you can 
> write valid strings using it, and it is likely programs that don't want 
> to test the strings will do exactly that.

The tag does not imply the string _is_ invalid, just that it _may_ be.

> "url" implies that all bytes with the high bit set, and many others, are 
> encoded with %. This is allowed but not emphatically not required! So I 
> am worried that "url" would be misleading.

Yes, "url" carries a lot of baggage, which is why I think that "!!
utf-url" is a bad idea (in addition to being too long and containing a
'-').

> This can be used to store invalid UTF-16, and even invalid UTF-32 
> (provided you extend UTF-8 to cover 32 bits).

AFAIK UTF8 covers 32 bits already (e.g., Perl's utf8 does), once you
discard the Unicode validity requirements.

> So there is an argument 
> that it should not imply UTF-8. However it certainly is using details of 
> UTF-8 in the definition, and also I can imagine somebody wanting a new 
> scheme for invalid UTF-16 (such as %NNNN) and wanting a different tag 
> for that with "16" in it.

I don't see much use case for this. We have a real-world need for a
mostly UTF8 + occasional binary data, as this is how the world is
(painfully) making the transition to Unicode. Generally speaking,
systems that use UTF-16 (Java, OSX HFS) tend to enforce Unicode validity
at some point so the need for "UTF16 + occasional binary data" should be
very rare. (What is _not_ rate is that a UTF16 application will need to
read a file containing !!utfu data.)

Still, names are best when they are descriptive, and !!utf-u (or !!utfu,
or even !!utf-url) does not convey the tie to UTF-8. Hmmm...

> Therefore I still like "utf8" or "utf-8" the best. But "utf-u" is ok.
> Also "utf" and "url" because they are short.

How about "!!utf8u"? It is perfectly descriptive, and not too long.
Speaking of length - since 99% of the strings in the world are short,
any reasonable emitter would not attach "!!utf8u" to a string that doesn't
actually require it. After all the main use case here is file names and the
like which are pretty short, so testing them should not be a big performance
issue.

> I consider it a requirement that "%%" be two '%' characters in a row. 
> Otherwise we will get an exponential explosion when combining this with 
> other systems that use "%%" to quote a '%'.

Not really; we convert % to %25. Using %% would just save one character,
not enough to worry about (% isn't that common, even in URLs). And you'd
break compatibility with URL encoding. I think that being compatible
with it will be a win - people are familiar with it, there's library
code to process it, etc.

> I also don't like throwing an error when there is no ambiguity in the 
> expected result. And I want to disallow the temptation to add more % 
> escapes (such as for invalid UTF-16). A new tag should be used for that.

I agree. You just tried to sneak one above, though (%%) :-)

> > The main question about UTF-16 here is: What should a Java library (or
> > any library that uses UTF-16 or UTF-32 natively) do when loading this
> > tag?
> 
> The caller *must* be able to detect lossy conversion and I would prefer 
> a design where it is hard for callers to ignore the error. Also I wish 
> to make it easy to implement the reader by calling a language's built-in 
> UTF-8->UTF-16 converter. Therefore what should be done:
> 
> 1. The library *must* throw some kind of catchable error if conversion 
> to UTF-16 and back to UTF-8 would be lossy. This means it throws an 
> error on any invalid UTF-8 byte sequence, and also on the encoding of a 
> high surrogate followed by a low surrogate.

If I understand correctly, your concern here is that round-tripping some
"%nn%nn%nn" will be emitted as "%nn" because the date went through UTF16
at some point. This is a valid concern.

So basically, what you are saying is to forget about trying to convert
this to UTF16/UTF32. You can _try_ to convert hoping !!utf8u was not
"really" needed in the first place (even if it used some %nn escapes).If
it _was_ needed - well then, you are SOL (get an error). Basically your
only "safe" option is to work with the byte array.

In this world, if the _application_ chooses to employ a lossy conversion
to UTF-16 (e.g., if a Java program interprets the string as a file name
and needs to stuff it into a String object, come hell or high water) -
well then, that's the application's prerogative and it has nothing to do
with _loading_ the string. The application can obviously do anything at
all to its input data (that's what applications are for after all :-).
And of course nothing prevents a YAML library from providing an API that
makes it easy for the application to do this - as long as it is
understood this is _processing_ (changing the data) rather than merely
loading it.

The down side of this is that you can forget about automatically coaxing
such !!utf8u scalars into native strings in Java/Python/etc. as part of
the normal _loading_ phase of !!utfu8 tags, even if they allowed for
"invalid UTF-16", because there's simply no way to make the conversion
lossless.

Interesting trade-off. I guess "safety comes first". It also saves us
from dealing with invalid UTF16 and UTF32. However, this means I would
*definitely* call the tag "!!utf8u" as it is _strongly_ tied to UTF8.

Have fun,

    Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july