Re: macho 0.4 released

Klaus Weidner <[email protected]> Mon, 10 Nov 2003 13:03:16 -0600
Newsgroups	gmane.lisp.clump
Message-ID	<[email protected]>
On Mon, Nov 10, 2003 at 10:07:18AM -0800, Miles Egan wrote:
> On Mon, 2003-11-10 at 09:58, Klaus Weidner wrote:
> > 	=?ISO-8859-1?Q?Fr=E9d=E9ric_Brunel?=
> 
> I must plead ignorance on the character encoding issues, especially WRT
> lisp.  Would this just be a matter of tagging the html with a character
> encoding attribute or do I need to write different character values?

You need to write different character values.

The right thing to do is Unicode, see the following article for a
good summary:

	http://www.joelonsoftware.com/articles/Unicode.html

If you just want to handle iso-8859-1, Unicode UTF-8 output is easy, and
that way you could extend it to properly handle other encodings in the
future.

I'm not sure where this is standardized, but the algorithm that mail
clients seem to agree on is to use the quoted-printable encoding shown
above whenever the original text of a mail header field contains
non-7bit-ascii characters. This is usually done on a word-by-word basis.

The algorithm needs to find encoded words in the header fields:

  [whitespace or start-of-line]
  	=? CHARSET ? ENCODING ? ENCODED-DATA ?=
  [whitespace or end-of-line]

CHARSET is a string such as iso-8859-1 (either case) indicating the code
page used for encoding this entry. I haven't seen utf-8 used here, but
it's entirely possible that clients might use that.

ENCODING is either "Q" for quoted-printable, or "B" for base64.

To undo quoted-printable encoding, replace occurences of
=[0-9a-fA-F][0-9a-fA-F] with the character coded by the two-digit hex
number.

Undoing base64 encoding is more complex than what I feel like typing
right now, ask Google. There are probably CL libraries to do that, and if
there aren't there should be :-)

If the coded character is in the 7bit ascii range (that'll happen even in
QP format because any occurences of "=" or "?" in the word will also be
encoded), just output the character, 

If it's something different, you need to convert it to multibyte UTF-8
encoding. If the input charset is utf-8, just output the characters one
by one. For iso-8859-1, the conversion table is easy. Juggle the bits as
follows:

  Input:  1abbbbbb
  Output: 1100001a 10bbbbbb

For example: the e-with-accent in Frederic's name is character 0xE9 in
iso-8859-1, and the two-char sequence 0xc3 0xa9 in UTF-8. In Perl (no
flames please), that's: chr(ord($1)>>6 | 192) . chr(ord($1)&63 | 128)

In the HTML output, put the character set marker at the start of the HTML
<head> section - it needs to be before any rendered text for obvious
reasons.

For Unicode UTF-8 output, use:

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">

Any non-Unicode-aware clients will at least display 7bit text correctly,
because the encoding for that subset is identical.

-Klaus