Re: macho 0.4 released

On Mon, Nov 10, 2003 at 02:49:34PM -0800, Jan Rychter wrote:
> If we do this, let's get this right -- the breakage associated with most
> US programs assuming that the world is ASCII and most European ones
> assuming that the world is ISO-8859-1 has already been around way too
> long.
[...]
> I'm afraid Unicode is the only way out here. I don't think the solution
> you have proposed is any more "correct" than just leaving everything
> unexpanded and pretending that only ASCII exists...

That's why I changed my mind and described in my other mail how Macho
could be enhanced with Unicode *output* support with little effort.

On the input side, it's a fact of life that very many people are
*sending* mail using ISO-8859-1 (and other) encodings, and as long as the
encoding used is specified properly (as it is in the =?ISO-8859-1?
format) and not just unmarked 8bit characters, that should be supported
if it's not too much work. It's very easy for 8859-1, and for other
encodings it would be a matter of collecting and using existing
conversion tables to convert to UTF-8. 

I'm *not* proposing blindly interpreting all input as iso-8859-1 8bit
characters.

The old adage "be strict in what you produce, but liberal in what you
accept" applies here [1], and we can't retroactively fix the content of
existing mail archives. The output should definitely be
standards-compliant Unicode and not yet another kludged 8bit format.

The change I proposed would probably only be a couple of lines of code to
get the basic framework in place, and the more labor-intensive extensions
would at least fit in if someone gets around to doing them. I guess it's
time to put my fingers where my mouth is and code this if nobody beats me
to it :-)

-Klaus

[1] That arguably led to major brain-damage in the case of web browsers
accepting invalid HTML, which led to sloppy pages, which turned HTML into
a de-facto poorly specified mess because now *all* browsers needed to
accept invalid input. But that's a different situation - the =?CHARSET
encoding *is* the standard for getting non-ascii characters into e-mail
headers.