Re: Invalid UTF-8

On Mon, 2009-08-24 at 18:17 -0700, William Spitzak wrote:
> 
> Oren Ben-Kiki wrote:
> > I think I did not understand your proposal.
> > 
> > Here is what I _thought_ you suggested:
> > 
> > 1 - When a YAML library that _writes_ a file,
> > _if_ given a UTF-8 string with invalid bytes,
> > it will escape them as "\XNN".
> > 
> > 2 - When a YAML library _reads_ a file,
> > _if_ it returns UTF-8 to its caller,
> > it will convert "\XNN" back to the original invalid UTF-8 bytes.
> > 
> > 3 - When a YAML library that _reads_ a file,
> > _if_ it returns UTF-16/UTF-32 to its caller,
> > it will treat "\XNN" to an error (throw exception or whatever).
> > 
> > This means that the same file will be read by one YAML library and be
> > rejected with an error by another. This is unacceptable.
> > 
> > Your post seemed to indicate you are proposing something else, but I am
> > not certain what it is. Can you explain exactly how you see "\XNN"
> > working in the above three cases?
> > 
> > Oren.
> 
> The above description matches exactly what I am proposing, except \XNN 
> only throws an error if arranged such that the result is not valid UTF-8.

Well then... this isn't acceptable. We want every YAML file to be
readable by any YAML library, even if it happens to use UTF-16 or
UTF-32. That is, we want to be sure that if a file is readable by a C
libyaml it would also be readable by a Java libyaml. We call this
"portability" and we take it seriously.

As for your use case (file names etc.) - representing stuff that _isn't_
Unicode but is _almost_ Unicode - there are other workarounds as I
described in the previous posts.

Have fun,

    Oren Ben-Kiki


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july