Re: New invalid UTF-8 proposal using tags and %nn

On Sep 4, 2009 6:00pm, Oren Ben-Kiki <[email protected]> wrote:
> On Fri, 2009-09-04 at 12:52 -0700, William Spitzak wrote:
> > I do not want a tag that implies the sting is "invalid", because you can
> > write valid strings using it, and it is likely programs that don't want
> > to test the strings will do exactly that.
> 
> The tag does not imply the string _is_ invalid, just that it _may_ be.


I think he meant the name used for the tag as some people were suggesting
things like "utf8-invalid", which I also don't like.


> AFAIK UTF8 covers 32 bits already (e.g., Perl's utf8 does), once you
> discard the Unicode validity requirements.


There's an interesting article on Wikipedia about this,
http://en.wikipedia.org/wiki/Utf-8. And it seems to me that that article has
changed recently, because I believe it used to treat UTF-8 separately from
the Unicode definition. UTF-8 was originally meant to handle other multibyte
character sets as well as Unicode (in fact, I don't know that Unicode even
existed at that time; they certainly weren't as predominant as they are
now). I've used the original encoding for other alphabets that could expand
to 32 bits, but usually didn't.


> > So there is an argument
> > that it should not imply UTF-8. However it certainly is using details of
> > UTF-8 in the definition, and also I can imagine somebody wanting a new
> > scheme for invalid UTF-16 (such as %NNNN) and wanting a different tag
> > for that with "16" in it.
> 
> I don't see much use case for this. We have a real-world need for a
> mostly UTF8 + occasional binary data, as this is how the world is
> (painfully) making the transition to Unicode. Generally speaking,
> systems that use UTF-16 (Java, OSX HFS) tend to enforce Unicode validity
> at some point so the need for "UTF16 + occasional binary data" should be
> very rare. (What is _not_ rate is that a UTF16 application will need to
> read a file containing !!utfu data.)


I mostly agree with you on this Oren, with one caveat.

I've already said that it seems to me that most "Unicode" processing is more
UCS-2 than UTF-16 as it doesn't actually take surrogates into account. It
treats each member of a surrogate pair as a separate character, rather than
treating the pair as a single character, a Unicode code point. I haven't
worked with Java in a while, and wasn't concerned about character encodings
when I did, so I can't speak to Java, but I've never seen .NET reject a
string with an invalid surrogate, or any other invalid character for that
matter. Anything that could be considered an error doesn't actually seem to
happen until you go to display the data or do something else that actually
needs to understand the meaning of the string as characters. In my work in
other environments, I haven't ever encountered anything in their APIs that
suggests they behave any differently.

>From that, it seems to me that someone might want to encode invalid UTF-16
data (because it was in the input), but it wouldn't be invalid bytes, so
much as invalid words (I'm using "word" to mean 16 bits as on an Intel
processor), which was implied by %NNNN, but is worth being explicit about,
since inserting random bytes (not words) is a recipe for disaster in UTF-16,
and not one that can be fixed by simply allowing the encoding of bytes.

Still, I think the topic for now should be the UTF-8 data type, and leave
the UTF-16 type as a future concern.


> How about "!!utf8u"?


I like that name well enough. I was thinking something along the lines of
"utf8e" myself, but was concerned that might conflict with some existing
standard or implementation (much like "utf8x" does).


> So basically, what you are saying is to forget about trying to convert
> this to UTF16/UTF32. You can _try_ to convert hoping !!utf8u was not
> "really" needed in the first place (even if it used some %nn escapes).If
> it _was_ needed - well then, you are SOL (get an error). Basically your
> only "safe" option is to work with the byte array.
> 
> In this world, if the _application_ chooses to employ a lossy conversion
> to UTF-16 (e.g., if a Java program interprets the string as a file name
> and needs to stuff it into a String object, come hell or high water) -
> well then, that's the application's prerogative and it has nothing to do
> with _loading_ the string. The application can obviously do anything at
> all to its input data (that's what applications are for after all :-).
> And of course nothing prevents a YAML library from providing an API that
> makes it easy for the application to do this - as long as it is
> understood this is _processing_ (changing the data) rather than merely
> loading it.
> 
> The down side of this is that you can forget about automatically coaxing
> such !!utf8u scalars into native strings in Java/Python/etc. as part of
> the normal _loading_ phase of !!utfu8 tags, even if they allowed for
> "invalid UTF-16", because there's simply no way to make the conversion
> lossless.
> 
> Interesting trade-off. I guess "safety comes first". It also saves us
> from dealing with invalid UTF16 and UTF32. However, this means I would
> *definitely* call the tag "!!utf8u" as it is _strongly_ tied to UTF8.


I agree with Oren. To my mind, this is entirely a library/application issue.
The YAML processor (usually in the form of a library), needs to cooperate
with the application on how data is returned, always, for any type of data.
How well it does so, particularly when dealing with invalid data, is what
distinguishes a "good" library from a "bad" one. This isn't part of the YAML
specification or the tag's definition (what I've been calling the data
type's definition or metadata, but now call "tag's definition" because we
are talking about how to define the type in the YAML tag repository). It is
specific to the platform and library and, consequently, so is the
appropriate handling of any bytes that are "invalid".

The tag's definition only needs to state what data is being encoded, the
canonical form of that data, and the possible presentation forms, with maybe
some notes on the preferred presentation. How the library actually provides
that data is beyond the scope of the definition. While the definition might
also recommend some options, I don't think that would be particularly
useful, myself. Library writers need to decide how to be most useful to
their clients within the context of the environment that the library is for.


So, to move things along (and you'll please forgive me if I overstep myself
a bit again), I think the three questions that need to be asked are:


1) How do we define type of data that the tag represents? Most of the
discussion has been about this, so... should be straight forward enough to
answer.

2) What is the canonical form used for that data? Key to how YAML works with
different data types.

3) What are the possible presentation forms of that data? We might also
suggest a preferred presentation or even comment that writers should not use
this tag for strings that are entirely valid Unicode.


And, of course, we need to settle on a name, but we seem to be coming to the
conclusion of that :-). After these questions are answered, then a
discussion of how PyYAML or some other implementation should expose this
data might be worth having. Myself, I don't care how PyYAML will do so,
which is why I'm trying to get off that topic. That, and I'm eager to see
the new type's definition.


If we do continue discussing the UTF-16 (utf16u or whatever), then we'd need
to answer the same questions for that type as well. I would suggest using
the term "word" instead of "bytes" though (along with a definition of "word"
since it varies based on the environment).


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july