utf8u tag proposal

Invalid UTF Type for YAML Version 0.1

Draft of proposal.

Author:
Bill Spitzak
<[email protected]>

Copyright 2009...

This document may be freely copied provided it is not modified.

STATUS

This specification is a draft reflecting consensus reached by members
of the yaml-core mailing list. Any questions regarding this draft
should be raised on this list.

----------------------------------------------------------------

URI:
        tag:yaml.org.2009:utf8u

Shorthand:
        !!utf8u

Kind:
        Scalar

Canonical:
        (%[0-9a-f][0-9a-f])*

Regexp:
        .* (any string is accepted!)

Definition:
        A sequence of zero or more bytes (8-bit values).

        The sequence "%xx" where x are hex digits (0-9,A-F,a-f)
        represents a single byte with the value equal to the hex
        number. "%20" is a single byte with value 0x20.

        A '%' not followed by two hex digits represents a single
        0x25 byte (the ASCII/UTF-8 value of '%') and parsing continues
        with the next byte ("%%20" is a 0x25 followed by a 0x20).

        All other characters are converted into their one to four
        byte UTF-8 representation.

Examples (these are all the same string):

        - !!utf8u A-acute is Á in Unicode
        - !!utf8u "A-acute is \xC1 in Unicode"
        - !!utf8u A-acute is %C3%81 in Unicode
        - !!utf8u
           "%41%2d%61%63%75%74%65%20%69%73%20%c3\
            %81%20%69%6e%20%55%6e%69%63%6f%64%65"

----------------------------------------------------------------

RATIONALE:

Many systems process UTF-encoded Unicode but ignore errors in the
encoding until display. This is often far easier and more reliable and
avoids security problems and/or DOS attacks. It also allows
conversions during display that are useful for the end user but would
be dangerous at any earlier stage (such as using a different
encoding).

Examples are Unix UTF-8 filenames, Windows UTF-16 in almost all apis
including filenames, %-encoded sequences in URLs, and many HTML
documents marked as UTF-8.

Experience has shown that if the storage damages *valid* UTF-8 or
makes it difficult to insert in a file, the user's response is usually
to treat the data as being in a different encoding such as
ISO-8859-1. Often this response is so common that interoperability
requires redefining the original data as being this other encoding.

Therefore we require a format that can store an invalid UTF array but
will read and write a *valid* encoding with as few changes as
possible, and utf8u is designed for this.

The format was chosen to match the %-encoding scheme used in URLs, and
is compatible with it. However utf8u also allows strings that are not
valid URLs: it can contain any Unicode character unchanged.

A '%' not followed by two hex digits is literal. This was chosen for the
following reasons:

1. Makes "%%", which is often used to escape a '%' in other systems,
work as a literal.

2. Minimizes the changes to valid strings.

3. Throwing an error when it is obvious and unambiguous what the
string means is not user-friendly.

4. Makes it impossible to "extend" this tag with new %-escapes. This
is a purposely-designed feature.

ENCODING NOTES:

The simples encoder would follow the "canonical" rules and replace
every byte with "%xx", however this will make the string unreadable in
the saved file and this is not recommended. An encoder should instead
do some or all of the following steps so as little of valid UTF-8 is
changed when written:

 1. Write all bytes without the high bit set except '%' unchanged.

 2. Any sequence of bytes that is a valid UTF-8 encoding is written as
the Unicode character (the UTF-8 encodings of U+D800..U+DFFF are *not*
valid and must remain written as three %xx sequences).

 3. If the two characters after a '%' are not hex digits then you do
not need to encode the '%' and can write it literally.

 4. If nothing except '%' needs to be encoded, then don't use the
utf8u tag at all and write the string as a normal YAML scalar.

DECODING NOTES:

You MUST check the two characters after a '%' and leave the '%'
unchanged if they are not a pair of hex digits. A '%' less than two
characters from the end of the string must also be unchanged.

Backslash removal is done by yaml before the decoding. Therefore:

1. "\x25\x32\x30" is a single 0x20 byte (because yaml converts the \x
sequences into "%20"). WRITERS SHOULD NOT RELY ON READERS DOING THIS
CORRECTLY!

2. "%5cx20" is the four bytes '\','x','2','0'. It is not a space
character. This is part of the canonical defintion so you must do it
correctly and writers can assume it.

----------------------------------------------------------------
UTF-16:

Because invalid UTF-16 (and any array of 16-bit words) can be
losslessly converted to "invalid" UTF-8, this may also be used to
store invalid UTF-16.

This may be a good idea rather than making yet another tag for invalid
UTF-16. In particular it means filenames and other text data may use
the same tag on both Unix and Windows (the fact that Unix can store a
superset of strings is irrelevant as programs must already deal with
this due to Unix accepting more punctuation marks).

Programs that wish to avoid dealing with UTF-8 can use these rules to
convert their UTF-16:

ENCODING NOTES:

All words other than the surrogate halves and '%' are encoded unchanged as a
Unicode character.

A '%' can be written as "%25". This is not required if the next two
characters are not hex digits.

A high+low surrogate pair (ie *valid* UTF-16) is encoded as the
matching Unicode character.

Each mismatched surrogate half is encoded as "%nn%nn%nn" where nn are
the hex values of the UTF-8 encoding of that word's value.

If there are no mismatched surrogate halves, the program may wish to
omit the tag and write the value as a normal scalar.

DECODING NOTES:

Decoder can use all characters except '%' unchanged.

It must accumulate up to 6 "%nn" sequences where n is a hex
digit. Decode this into up to 6 values, which I will call "bytes" for
obvious reasons. Notice that non-%-sequences do not need to be merged
in any way.

If the leading 1-4 "bytes" are a valid UTF-8 encoding then replace that
portion with the matching Unicode character, and start again after it.

If the leading 3 "bytes" are the UTF-8 encoding of a low surrogate half
U+DC00..U+DCFF then replace that portion with the surrogate half.

If the leading 3 "bytes" are the UTF-8 encoding of a high surrogate half
U+D800..U+DBFF *and* there are *not* 3 bytes after it that are the valid
encoding of a low surrogate half, then replace that portion with the
surrogate half.

All other byte sequences should trigger an error, of similar severity
as libyaml failing to parse the file. The string could not possibly
have been a UTF-16 array originally. However it may be useful for the
converter to provide access to a "sanitized" converted string after
reporting the error. This sanitized string should contain the
following conversions:

The 6 "byte" sequence of a matching hi+low surrogate pair is converted
to the matching hi+low surrogate pair and thus a single Unicode character.

All other sequences convert the first "byte" to 0xDCxx and then
continue decoding after it.

----------------------------------------------------------------
UTF-32:

It is also possible that this can write invalid UTF-32.

This could be done by extending the definition of UTF-8 to 32 bit
numbers, and then working similar to the UTF-16 description above,
although it is much easier as surrogate halves can always be decoded.

The original UTF-8 was defined up to 31 bit numbers and there are two
possibilities for 32-bit numbers:

1. Use both 0xFE and 0xFF followed by 5 trailing bytes.
2. Use 0xFE followed by 6 trailing bytes (allowing up to 36-bit numbers)

In any case until an actual use case comes up this is probably a minor
consideration.