Re: Invalid UTF-8

The main issue here appears to me to be over whether the handling of
invalid UTF-8 is a display issue.

I argue that it is not a display issue for this simple reason: the
existence of methods which operate on UTF-8 which have nothing to do
with display.

Let's look at the effects of dealing with it as a display issue.
Allowing invalid UTF-8, or allowing a special encoding on top of
UTF-8, requires that all methods which operate on the UTF-8, and
therefore need to parse it in some way, need to be aware of this extra
possible syntax to avoid mangling the string's semantics. You would be
effectively moving the burden of data entry issues deeper down the
stack. That would not help UTF-8 awareness at all. It would do the
opposite.

To put it simply: it is not a display issue, but rather it is a data
entry issue.

The solution can be simple: You sanitize all user-submitted syntax
before you send it to a library which operates on that syntax. This
can be as easy as offering a preview functionality, and stripping out
invalid UTF-8 or ASCII sequences.

I had to deal with an even worse version of this problem when
providing an interface for users to edit php smarty templates. The
smarty would get compiled to php, and smarty syntax errors would
sometimes result in uncatchable fatal php errors. We simply provided a
preview functionality, and the problem went away.

However, you do have a secondary point I would like to address. Most
applications don't do much UTF-8 manipulation, and just use decode().
This is why it appears to be display issue, and why it is
understandable and perfectly valid that you would want a library that
is capable of handling invalid syntax easily.

The problem is that the libraries then become no longer suitable in
software that actually does a lot of UTF-8 manipulation. You
effectively move the burden of data entry issues to that class of
programs, which now need to be intimately aware of the error handling
properties of the library.

Why not just avoid that problem altogether and write a little wrapper
library which handles data entry issues the way you want, instead of
trying to fundamentally change the nature of the library? That would
be a handy tool which a lot of projects may use. It could even have
its own library for solving the data entry problem on other tools, and
your cause could move forward.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july