problems with 8bit headers and html mail

Hi,

I'm new to the list, but I've spent my last two weeks importing my mail
archive into ZOE. I like it a lot, but the conversion ran into a lot of
troubles.

It's because I, as a Taiwanese, have a lot of mail in "big5",
a charset commonly used for traditional Chinese, and some in UTF-8 or
other DBCS (double-byte character sets for East Asian languages). Many
of them couldn't be imported properly. I managed to solve most of the
issues (I'll contribute to the ZoeDoc wiki if it's of value to others),
but I need help with two problems.

The first is about 8bit characters in headers. When Zoe receives a
message with 8bit headers and content (big5 characters sent verbatim,
not encoded), the content is displayed correctly (if the "charset" is
properly set), but the headers ("From" and/or "Subject" fields in
Chinese) are not.

I know 8bit headers are against RFC rules (right?), but they're very
commonly used in DBCS world. Since the content part is displayed fine, I
believe ZOE does know how to handle such characters properly. The
problem is (my guess), ZOE assumes the headers use only 7bit characters.
It would be nice if ZOE can apply the information of "charset" to the
headers, if it sees 8bit characters there. In other words, please treat
8bit characters in headers the same way the content is treated. That's
what all the email clients I've tried (Becky, Thunderbird, The Bat!, and
Outlook Express) do.

This is important for 8bit headers are still in use today by a lot of
people. When importing such messages, my workaround is to redirect them
using Becky, which is the only mail client I know that would re-encode
headers when redirecting mail. All the others would simply add those
"resend" headers and leave the original message unchanged.

The workaround is time consuming, for Becky can redirect messages only one
at a time, asking for a recipient each time, but that's not my biggest
problem. With new mail, Zoe retrieves a message often before I have a
chance to do this trick. With the "delete duplicates" option enabled,
the redirected message won't get in before I delete and purge the
original one. (Can't turn off the option, for ZOE archives both my
wife's and my mail, and we do have quite a few duplicates.)

Enough for the first one. The second problem is with html-only mail in
Chinese that uses 8bit content transfer encoding. With such mail, even
when the charset is correct, the content isn't properly displayed. ZOE
has no problem displaying multipart/mixed messages with both a plain
text part and an html part. But with a pure html message, it generates a
text part without taking into account the charset setting (again, just a
guess).

My workaround was to export such mail to a text file, and manually
created a plain text part by cleaning up the html codes in a text editor.
Reimport it into my mail client, and then redirect or forward it. The
trick not only is time-consuming, but also fails with new mail, for the
same reason stated above.

Again, such mail is common place today, so a long-term solution is
needed. I'm not a Java coder, but am otherwise willing to provide any
help I can.

Finally, thanks again for such great software. Despite all the troubles,
I think the effort is worthwhile -- if Zoe can handle new mail properly.

-- 
Best regards,
Ming-Li



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl