Re: Invalid UTF-8

On Thursday 20 August 2009 5:57:56 pm Ben Woolley wrote:
> The main issue here appears to me to be over whether the handling of
> invalid UTF-8 is a display issue.
>
> I argue that it is not a display issue for this simple reason: the
> existence of methods which operate on UTF-8 which have nothing to do
> with display.
>
> Let's look at the effects of dealing with it as a display issue.
> Allowing invalid UTF-8, or allowing a special encoding on top of
> UTF-8, requires that all methods which operate on the UTF-8, and
> therefore need to parse it in some way, need to be aware of this extra
> possible syntax to avoid mangling the string's semantics. You would be
> effectively moving the burden of data entry issues deeper down the
> stack. That would not help UTF-8 awareness at all. It would do the
> opposite.
>
> To put it simply: it is not a display issue, but rather it is a data
> entry issue.
>
> The solution can be simple: You sanitize all user-submitted syntax
> before you send it to a library which operates on that syntax. This
> can be as easy as offering a preview functionality, and stripping out
> invalid UTF-8 or ASCII sequences.
>
> I had to deal with an even worse version of this problem when
> providing an interface for users to edit php smarty templates. The
> smarty would get compiled to php, and smarty syntax errors would
> sometimes result in uncatchable fatal php errors. We simply provided a
> preview functionality, and the problem went away.
>
> However, you do have a secondary point I would like to address. Most
> applications don't do much UTF-8 manipulation, and just use decode().
> This is why it appears to be display issue, and why it is
> understandable and perfectly valid that you would want a library that
> is capable of handling invalid syntax easily.
>
> The problem is that the libraries then become no longer suitable in
> software that actually does a lot of UTF-8 manipulation. You
> effectively move the burden of data entry issues to that class of
> programs, which now need to be intimately aware of the error handling
> properties of the library.
>
> Why not just avoid that problem altogether and write a little wrapper
> library which handles data entry issues the way you want, instead of
> trying to fundamentally change the nature of the library? That would
> be a handy tool which a lot of projects may use. It could even have
> its own library for solving the data entry problem on other tools, and
> your cause could move forward.
>
>

+1 
It is a data issue, pushing it up the stack only prolongs the agony.


-- 
Adrian Klaver
[email protected]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july