Re: Invalid UTF-8

Ben Woolley wrote:
> The main issue here appears to me to be over whether the handling of
> invalid UTF-8 is a display issue.
> 
> I argue that it is not a display issue for this simple reason: the
> existence of methods which operate on UTF-8 which have nothing to do
> with display.
> 
> Let's look at the effects of dealing with it as a display issue.
> Allowing invalid UTF-8, or allowing a special encoding on top of
> UTF-8, requires that all methods which operate on the UTF-8, and
> therefore need to parse it in some way, need to be aware of this extra
> possible syntax to avoid mangling the string's semantics.

I'm sorry but you have EXACTLY BACKWARDS!! Detecting errors requires 
understanding the "semantics". Treating the UTF-8 as an array of bytes 
does not, but it means encoding errors are passed through!

UTF-8 is TRIVIAL to process if you stop being numbed by ASCII history 
and realize that "characters" are not very important. Even with errors 
it is self-synchronizing, a pattern will not match except at a character 
boundary.

I don't know what to do, really. For some reason UTF-8 turns otherwise 
intelligent programmers into complete morons. All I can suggest is you 
try to write some code that works with UTF-8 instead of blindly calling 
decode(). And check what you are doing about mismatched UTF-16 
surrogates. Nothing? And your program still works? But you claimed right 
above that ignoring these such will require "special encoding atop 
UTF-16 and therefore the need to parse it in some way". I hope you can 
realize you are talking total nonsense when you realize that the 
solutions for UTF-8 and UTF-16 are IDENTICAL!!!

 > You would be
> effectively moving the burden of data entry issues deeper down the
> stack. That would not help UTF-8 awareness at all. It would do the
> opposite.

WRONG!!!! Throwing errors is exactly what is damaging UTF-8 awareness. 
The reaction by a programmer, IN ONE HUNDERED PERCENT OF THE CASES I 
HAVE SEEN, is to change the encoding to ISO-8859-1 or to strip all bytes 
with the high bit set. This includes a poster right here who said that 
the solution for Unix filenames is to restrict them to ISO-8859-1. This 
has happened and is happening, over and over and over again, right now, 
destroying internationalization! Anybody doing this is part of the 
problem and is very annoying that you would make the claim you are helping.

> The solution can be simple: You sanitize all user-submitted syntax
> before you send it to a library which operates on that syntax. This
> can be as easy as offering a preview functionality, and stripping out
> invalid UTF-8 or ASCII sequences.

But I can't use libyaml to do this "sanitize" without these changes! IT 
THROWS UNRECOVERABLE ERRORS and I can't "strip out invalid UTF-8"!!!

> I had to deal with an even worse version of this problem when
> providing an interface for users to edit php smarty templates. The
> smarty would get compiled to php, and smarty syntax errors would
> sometimes result in uncatchable fatal php errors. We simply provided a
> preview functionality, and the problem went away.

Oddly enough you got these PHP errors despite having made sure all your 
UTF-8 was valid? What a big help that was! I think this is an excellent 
counter point to your arguments. Removing some small set of possible 
errors from the data stream is useless and simply a waste of time!

> However, you do have a secondary point I would like to address. Most
> applications don't do much UTF-8 manipulation, and just use decode().
> This is why it appears to be display issue,

Again totally backwards from my understanding. The reason people DON'T 
believe it is a display issue is that they are used to decode() throwing 
the error. decode() is NOT a display function.

If it was a "display issue" there would be a "draw this utf8 string" 
api, not "decode".

> Why not just avoid that problem altogether and write a little wrapper
> library which handles data entry issues the way you want, instead of
> trying to fundamentally change the nature of the library? That would
> be a handy tool which a lot of projects may use. It could even have
> its own library for solving the data entry problem on other tools, and
> your cause could move forward.

This is EXACTLY what I am trying to do. I'd like you to explain how I 
can use libyaml to do this if it craps out with an error and does not 
give me the invalid UTF-8? Am I supposed to use the block read/write api 
rather than the FILE api and "sanitize" my UTF-8 there, while somehow 
not screwing up the quoting? And strip out my encryption after I get the 
libyaml scalars? Do you know how incredibly complicated and slow that 
will be? Do you know that the error messages for libyaml won't give me 
the correct line and charcter numbers any more?

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july