Re: Invalid UTF-8

Ben Woolley wrote:
> On Fri, Aug 21, 2009 at 3:51 PM, William Spitzak<[email protected]> wrote:

>> Detecting errors requires
>> understanding the "semantics".
> 
> I don't know what you mean here. Please clarify.

Here is strcpy without detecting errors, it does not contain any 
"semantics" other than "nul terminates the string":

	while (*d++ = *s++);

Here is strcpy WITH detecting errors, it does contain some more 
"semantics" such as "what makes a valid UTF-8 encoding":

	while (*s) {
           if (!validUTF8(s)) throw error;
	  *d++ = *s++;
	}
	*d++ = 0;

> You seem to want all UTF-8 parsers to be validating parsers. But at
> some point, you need to declare a point of responsibility for
> transposing the data. That is certainly not in a serialization library
> that does its own transposing. Let it do its job, and just its job.

I don't understand this. I in fact want to *remove* validation from the 
parsers, not add it!

>>> However, you do have a secondary point I would like to address. Most
>>> applications don't do much UTF-8 manipulation, and just use decode().
>>> This is why it appears to be display issue,
>> Again totally backwards from my understanding. The reason people DON'T
>> believe it is a display issue is that they are used to decode() throwing the
>> error. decode() is NOT a display function.
>>
>> If it was a "display issue" there would be a "draw this utf8 string" api,
>> not "decode".
> 
> Where, in all of this, is the "draw this kinda-UTF-8 string" API?

I meant a function that takes UTF-8 on input and draws it on the screen. 
It is not relevant whether or not it accepts invalid UTF-8. Such a 
function I would call a "display" function. A converter to UTF-16 is NOT 
a "display" function, since you are not required to draw the resulting 
string.

>>> Why not just avoid that problem altogether and write a little wrapper
>>> library which handles data entry issues the way you want, instead of
>>> trying to fundamentally change the nature of the library? That would
>>> be a handy tool which a lot of projects may use. It could even have
>>> its own library for solving the data entry problem on other tools, and
>>> your cause could move forward.

Because the easiest way to do this is to replace libyaml entirely. Using 
the block api with some kind of filter function, escaping all invalid 
utf-8 and also escaping the escape sequence, and removing this on 
return, and fixing the column numbers in error messages, while 
technically possible, is insane, it is complicated, slow, and impossible 
for anybody to further alter yaml syntax.

We will be using a forked version with this patch for now.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july