Re: utf8u tag proposal

Hi William,

>> Hence, we should not serialize our data
>>
>>  [ "a%b%c", "d%e%f" + badcode, "g%g%i" ]
>> as the next.
>>
>>  - !!utf8u a%%b%%c
>>  - !!utf8u d%%e%%f%XX%XX   # %XX%XX denotes the bad code
>>  - !!utf8u g%%h%%i
> 
> I want to make clear that I ABSOLUTELY DO NOT WANT "%%" to be the escape 
> for '%'. The escape is "%25". I want the definition to say that '%' 
> followed by anything other than 2 hex digits is a literal '%', this is 
> on purpose so nobody is tempted to say "%%" is an escape! So these 
> examples are written like this:
> 
>  - !!utf8u a%25b%25c
>  - !!utf8u d%25e%25f%XX%XX
>  - !!utf8u g%25h%25i

You are right.
I had to write this.

> The output converter can also check if literal percent works and output 
> this, this is my recommendation:
> 
>  - !!utf8u a%b%c
>  - !!utf8u d%e%f%XX%XX
>  - !!utf8u g%h%i

In my opinion, this is not sperior to the former.
BlueG's example "%cat" gave me this conclusion.


> The reason for this is so that a YAML program that ignores the !!utf8u 
> tag will get as close as possible to the original string, and so it is 
> fairly easy using a text editor to remove/replace the bad codes and the 
> tag and turn the bad string into a good one.

I point out that no YAML program will ignore any unknown Tag. 
They will just reject a document with unknown node. So, I think
it will not be so benefitial to have !!utf8u as close as
possible to the original string.

In addition, it seems easier for me to convert all "%25" to "%" 
with a text editor than to distinguish "%cat" from "%dog" by my 
eyes.

So, I vote one to the form "!!utf8u a%b%c" as the canonical form. 
At the same time, I think that a parser can also accept 
"!!utf8u a%b%c" as "a%b%c" because there is no uncertainty about 
such a flexible interpretation.


>> Then, the library seamlessly converts the !!utf8u nodes back into 
>> string variables when deserialize.
> 
> This probably is not necessary. It is a lot easier for the application 
> to look for the tags and decode them. Also there seems to be an absolute 
> insistence that only valid utf-8 strings be returned by the library 
> which seems to make any such idea unacceptable to some here.

I agrre with you in part. 
Let me study the use cases, a little more.

Imagine my ruby application is using the YAML library. 
With the library, I can easily convert my Array of String to 
a YAML document.

  # Build an array of string
  array_of_string = []
  array_of_string << input.get_next() while input.any_more?()

  # Serialize!
  yaml = YAML.dump(array_of_string)

Usually, I can convert the YAML document back to an Array of 
String.

  # Deserialize!
  restored = YAML.load(yaml)

But, when I had a bad string in the input, it is now encoded
in a !!utf8u node. Then, the restored array will include some 
YAML.Utf8u class objects.

I will realize to have such an object when I invoke a method 
of String that is not implemented by Utf8u class.

  restored.each do |s|
    s =~ /????/u  # raises a method missing error
    ...
  end

Or, more ideally, I should confirm that the types of the restored 
objects are really what I'm expecting, before I use them. This is 
also better for an application that does not know anything about 
!!utf8u. Then, a document with !!utf8u nodes is safely *rejected* 
before causing a run time error.

  def check_restored(restored)
    return false unless restored.is_a?(Hash)
    restored.each do |s|
      return false unless s.is_a?(String)
    end
    true
  end

To be compatible with !!utf8u, I should modify this function as
the next.

  def check_restored(restored)
    return false unless restored.is_a?(Hash)
    restored.collect! do |s|
      if s.is_a?(String)
        s
      elsif s.is_a?(Utf8u)
        print "Warning: invalid utf8 string." unless s.is_valid_utf8?
        if NEEDS_INVALID_UTF8_AS_IS
          # accept the value
          s.value
        else
          # somehow convert it to a safe string
          s.convert_to_safe_utf8
        end
      else
        # invalid node found
        return false
      end
    end
    true
  end

This will work perfectly. A !!utf8u node is converted back to
a string either as is or with some more sophisticated way.
I guess this is what William supposed.


On the other hand, when I deserialize the document with some 
schema-oriented system, the use case is different because in 
such a system, Array that contains any Utf8u object can not be 
accepted by the schema. For example, in C#, we can not assign
Utf8u class object in a string variable. So, unless the library 
has some call back to allow users to convert !!utf8u to string 
*BEFORE* applying the schema, I can never accept the document.
Here, I assume the YAML library for ruby accepts a schema in the
next way.

  # the schema only accepts an Array that contains String
  schema = build_schema()

  # this will throw an invalid format exception if the
  # document contains some !!utf8u nodes
  restored = YAML.load(yaml, schema) 

So, before invoking YAML.load, I have to set the callback.

  # a filter function is expected to return an Array
  # [Bool whether_node_is_processed, Object deserialized_object]
  #
  def convert_utf8u_node_to_string(node)
    # leave the node unprocessed unless it has !!utf8u tag
    return false unless node.Tag == "!!utf8u"

    print "Warning: invalid utf8 string." unless Utf8u.is_valid_utf8?(node.Value)
    if NEEDS_INVALID_UTF8_AS_IS
      # accept the value
      return true, node.Value
    else
      # somehow convert it to a safe string
      return true, Utf8u.convert_to_safe_utf8(node.Value)
    end
  end

  YAML.deserializing_filters.add( new Proc { |node| filter_yaml_node(node) } )

  # the schema only accepts an Array that contains String
  schema = build_schema()

  # now we can accept a YAML document with some !!utf8u nodes
  restored = YAML.load(yaml, schema) 

Instead of doing this, I might prefer the library to provide some 
option with which I can choose the way how I accept !!utf8u. 

  YAML.options.utf8u_deserializing =
    :pass_through | :as_string | :as_safe_string

I wonder if it is also good to have such option even when the 
system is not schema-oriented.

Best,
Osamu TAKEUCHI

------------------------------------------------------------------------------
Come build with us! The BlackBerry&reg; Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9&#45;12, 2009. Register now&#33;
http://p.sf.net/sfu/devconf