Re: Invalid UTF-8

William,



I had been considering getting involved in this conversation and then
thought better of it, but I'll comment anyways to see if I understand...



William Spitzak wrote:

	What I need is a lossless way to store invalid UTF-8 in a scalar...



So the goal is to store data in a scalar that isn't a valid Unicode string.
In fact, this can already be done, but what's being discussed is having a
standard way of escaping invalid bytes. Based on other elements of the
conversation, it seems to me that the reason this ability is desired is so
that the invalid data can be stored as it travels from one system (or
program) to another in a way that is still readable by the user while it is
in its intermediary format.

In other words:

At the transmitter: The transmitter has data (a string, but not a valid
Unicode string) that it wants to store as a scalar. Examples of where this
data might come from include such things as filenames that were copied and
pasted before being transmitted or user text with an unexpected encoding.
The key is, this happens before transmission (before the YAML stream (or
file) is produced); this isn't the user directly modifying the YAML stream
(or file). If I'm wrong about that, then I've misunderstood. The transmitter
then produces the YAML stream (or file) with this invalid data encoded in
the scalar value; how exactly it is to be encoded is one subject of the
debate.

During transmission (or storage): The transmission (or storage) format is a
YAML document that is a valid UTF-8 (or 16, or 32) stream (or file), with
the invalid data safely encoded using an escape sequence so that while the
scalar value represented might not be a valid Unicode string (since Unicode
has a several rules for how strings are encoded that go beyond representing
a single code point), its representation within the YAML stream (or file) is
still valid UTF-8 (or 16, or 32) because the invalid data has been escaped.
In no way is this file modified during this stage so that it might contain
an invalid UTF-8 (or 16, or 32) byte sequence. So, if someone opens it in a
text editor, we are assuming at this stage that they then save it as a valid
UTF-8 (or 16, or 32) file and it is still able to be read as such. This
again because, although the values represented by scalars might be invalid,
their scalar representation in the byte stream (or file) is still valid. I'm
assuming this because of the comment about readers in your last e-mail
(sorry for not quoting it here).

At the receiver: The receiver reads in the byte stream, which is valid UTF-8
(or 16, or 32), and decodes the scalar, which is not necessarily a valid
Unicode string, but represents the data (a string, but not a valid Unicode
string) as it originally existed at the transmitter (plus any modifications
made during transmission). Key point: the byte stream was valid UTF-8 (or
16, or 32), but the value of the scalar (not its representation in the
stream) might not be.



So, if I've understood correctly, the issue is ~not~ how to deal with an
invalid stream (or file) at the byte level, but being able to have data that
is not a valid Unicode string encoded within a scalar. William, is that your
only concern? Or have I missed something? (because a lot of other things
have been discussed)



If it is your only concern, then it seems to me that the discussion becomes
centered around three questions (which, I think, have all been asked, but
which don't involve the byte encoding of the stream or file):

1) How can arbitrary bytes be represented within a scalar? As an aside, are
we discussing any kind of scalar or only double quoted scalars (which
support escape sequences)?

2) What data type is represented by the scalar? Note that it can't be the
normal YAML string because the meaning assigned to that data type is that it
is a valid Unicode string. Changing that would be a breaking change, so the
scalar would need to be representing some other data type (a less rigid type
of string, such as a byte string in Python).

3) What is the canonical form of that data type? This must also be a Unicode
string and is needed for equality testing.



Now, if the concern were just how to transmit the invalid data, then this
could all be accomplished using a new data type with a format that supports
the encoding of raw bytes. The data type, would again, have to be something
other than the normal YAML string, but it would still be stored as a mostly
readable scalar in the YAML file. Yaml already supports doing this.

The conversation, however, has focused on how to add a way to encode those
raw bytes in YAML and I can see how that might be helpful as it would allow
data from non-Unicode sources to be encoded, transmitted or stored in a
readable format (with the exception of possibly a few characters), and then
decoded back to its original form, using a standard escape sequence. A large
part of this challenge is met by defining an appropriate data type, which
can be assigned to a node using a tag and, in fact, there's no reason a data
type can't define escape sequences to use for this purpose. Of course, those
escape sequences would be interpreted after the standard escape sequences
(which means that if the backslash was used, it would need to be doubled)
and it sounds like what you would like is a standard escape sequence that
could be used instead.



Have I got this right so far?



If so, then with a new data type in hand that can handle strings that aren't
necessarily valid Unicode strings, all that's left is how to encode those
raw bytes in a scalar. A scalar's representation, of course, must be valid
Unicode even if its value is not, hence the need for escaping. Because of
the distinguishment in data type, a YAML reader can still treat as invalid
any scalar that has an invalid sequence of such bytes, since it knows from
the type (as determined by the schema and tag or lack thereof) that that
scalar must be a valid Unicode string (just as a float knows what scalars
are valid representations of floats) while still allowing such a scalar to
represent the new data type (or an even more arbitrary data type such as a
binary), which can handle the raw bytes.

Now, again, we're talking about a standard escape sequence used for this
purpose that the reader treats as part of the scalar's value, but that's
where things get sticky, because that leads to the discussion that, it seems
to me, has only been touched upon a few times. Because for that to happen a
number of questions need to be answered about how the reader would handle
those escape sequences to create the scalar's value.



If I'm correct up to this point, then it seems that that (how the reader
handles those escape sequences) is the only issue left to discuss. Well...
that and whether or not this is something that should actually be a part of
the standard, but there needs to be some idea as to what it would look like
and how it would work first.

But, I'm out of time. And I think I've bored everyone with my long summary
anyways (my apologies to you all). Let me know if I have the basic issue
right, William. If so, I have an idea that might help and could be used in
other contexts as well.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july