Re: New invalid UTF-8 proposal using tags and %nn

On Thu, 2009-09-03 at 19:08 -0400, BlueGM wrote:
> Yes, I think that would be an excellent first step. Having a new data
> type that is recognized as being a string, and looks like a string
> within YAML when it is composed entirely of valid characters, but
> which can accept invalid bytes allows the representation of strings
> from many sources that might contain invalid byte sequences. The
> canonical values (not the value of the data type or its scalar
> representation, but the representation that YAML uses when testing for
> equality) might need some normalization, but deciding that is part of
> defining the new data type.
>  
> I do disagree with having the % sign only apply if the next two
> characters form a hexadecimal digit less than 0x80. My reason for this
> objection, though, is for unquoted scalar values, which don't have a
> native way to escape characters. I don't see a reason to deny them
> byte values below 0x80 if invalid bytes are being escaped anyways, but
> this is not a major limitation and I assume you have a reason.
>  
> Moreover, though, once its known that we are not talking about the
> existing YAML string type, but a different string type, the limitation

I've been watching this thread for a while. Here are some of my
understanding of the issue, which might be helpful or might be wrong.

My basic understanding is of string and encodings is that 
String differs from its representations. 


The representation of a string can be a Unicode byte array, a Utf8 byte
array, or an index value from a list(eg, GQuark). However in practice a
string always has to be represented in one way or another. An invalid
utf8 byte array doesn't represent any string, by the definition of
('invalid')

May confusion arises, these two terms should be clearly distinguished in
the context.

The two questions we had can be rephrased into:

Q1. whether 'the string of the YAML document' shall be represented by a
byte array that can be either valid utf8 or invalid utf8
Q2. Is it possible to represent an invalid utf8 byte array in a YAML
document string. 

The answer to Q1 was NO. William's proposal was to dress Q2.

However if the proposal is about representing an invalid utf8 byte array
in a YAML document(which itself is always a valid utf8 byte array as it
is a representation of a string), then I don't think !!utf8 as a
suitable name for the data-type. The data-type is a container of both
valid and invalid utf8, regardless the name 'utf8' strongly suggests
'valid' utf8.

I don't have any better name in mind though. 'bytestream' is also a
horrible name, because the bytes in the stream are refined to valid utf8
sections and invalid utf8 sections, which makes it different from any
bytestreams. 

What about something like '!!bytestream-utf8' or '!!inline-bytes'? 


Regards,

Yu


>  that the value being decoded must be a valid Unicode string to avoid
> breaking existing applications disappears. The new string type
> wouldn't have that contractual obligation to fulfill. Then it's worth
> asking the spec writers how the new data type's scalar representation
> can be made more efficient or aesthetic in a future version of YAML.
> And I have some ideas about that:
>  
> 1) To address the tag issue, all that would be needed is a way to say
> scalars default to the new data type rather than to YAML strings. This
> is something I would like for other reasons and could perhaps be done
> with a directive. It also opens up the possibility for others to
> establish their own default data type for scalars. If the override
> were based on the type of scalar (quoted, plain, or block), then it
> would also be possible to do something like the following:
>  
> ----------------------------------------
> %DEFAULTBLOCKSCALAR yaml.org,2002:binary
>  
> MyFirstIcon:
>     fe8e0fp3
>    e384kc7e
>    dwe38c3k
>    3kd83kf8
>  
> MySecondIcon:
>    asei3kdu
>    ade38bk7
>    3jf83k59
>    38vc2u5a
> ----------------------------------------
>  
> Of course, in this example, it isn't that big a deal to add the !!
> binary tag, but that's the basic idea (if a bit crude... %
> DEFAULTBLOCKSCALAR is rather wordy)
>  
> My own reason for wanting such a tag would look something more like:
>  
> ----------------------------------------
> 
> %DEFAULTBLOCKSCALAR mycompany.com,2009:wiki
>  
> Room1:
>    Name: Great Foyer
>    Description:
>       .
>       You are standing in the great foyer. All around you
>       are white marble statues. A warm fire flickers in the
>       fireplace.
>  
>       There are doors to the north, south, and east.
>       .
> ----------------------------------------
>  
> That doesn't illustrate why I suggested overriding based on the type
> of scalar, however. The observant among you will have noticed that
> the . has meaning. And my actual usage is far more complex using a
> Wiki style markup (and not for a text game), but you get the idea.
> You'll also note that its meant to be a custom data type, not a
> standard one.
>  
> And, for something similar to your problem (assuming the utf-8 data
> type gets defined):
>  
> ----------------------------------------
> %DEFAULTPLAINSCALAR yaml.org,2009:utf-8
>  
> - Filename: My First File
>   Action: compile
>  
> - Filename: My Second File
>   Action: copy
>  
> - Filename: My%82Third File
>   Action: compile
> ----------------------------------------
>  
> 2) My second idea, another way to handle the tag issue, would be to
> allow one of the special characters to be assigned as a shortcut for a
> complete tag using something similar to a preprocessor directive
> again. It would look something like the following:
>  
> ----------------------------------------
> %ATTYPE yaml.org,2009:utf-8
>  
> - Filename: My First File
>   Action: compile
>  
> - Filename: @ My Second File
>   Action: copy
>  
> - Filename: @ My%82Third File
>   Action: compile
> ----------------------------------------
>  
> In my example, the %ATTYPE directive is assigning the @ sign to be a
> shorthand for the yaml.org.2009:utf-8 tag (which could also be
> written !!utf-8 if the !! prefix hasn't been reassigned).
>  
> 3) My third idea isn't actually my own... its yours. Once it's been
> established that we are dealing with a data type other than the
> default string, there's no reason the scalar's value has to be a valid
> Unicode string (again, because it isn't breaking any contracts... it
> was always meant to have some invalid bytes). Of course, the details
> of how it is handled in each step of the YAML processor would still
> need carefullly considered, but we'd know that the data is valid on
> both ends of that process. It is validly encoded within the YAML
> stream (or file) and it a valid string as far as the application is
> concerned (because its type warned the application that it wasn't
> necessarily a valid Unicode string). That puts us a lot closer to
> finding something that works for everyone (well... at least more of
> us). In talking about how to encode such a data type, rather than
> a normal YAML string, I don't think you would encounter so much
> resistance. Even without a native escape sequence though, having a
> standard data type that can store a string that might contain invalid
> bytes will give libraries and applications a way to encode that data
> in a way that is still human readable.
>  
>  
> The one big caveat to a new data type is that the application needs to
> know how to deal with it, but, actually, that isn't so hard. In
> Python, the most appropriate action for a python processor may be to
> return a byte string (old style string, I think its called?) if the
> scalar is of the new data type and a unicode string otherwise. In this
> case, the application using the processor would just get strings of
> one kind or the other. Or, a smart library writer (in Python or any
> other language) might expose an option so that both types are returned
> by the processor using the same data type as much as possible. Then,
> the application wouldn't need to deal with it as anything other than
> an invalid scalar if it didn't convert. Either way, the necessary
> signal is there for the application to decide what to do with it (most
> likely, using options in a library).
>  
> Now, for those who might wonder why not just use a binary, I think
> William was making an excellent point about its human readability. Why
> obscure the meaning of the entire message for just a few bytes? But,
> if your application demands that you can't lose those bytes, then you
> need some other alternative. I would add to this that it serves as a
> signal to the application and processor that the data is supposed to
> be a string, if they are capable of representing/handling it in that
> way. Otherwise, they can treat it as a binary, or whatever. The intent
> is known so the library and application can decide.
>  
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with 
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________ Yaml-core mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/yaml-core


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july