Re: problems with 8bit headers and html mail

Hi Ming-Li,

On Oct 25, 2004, at 21:05, Ming-Li wrote:

> I'm new to the list, but I've spent my last two weeks importing my mail
> archive into ZOE. I like it a lot, but the conversion ran into a lot of
> troubles.
>
> It's because I, as a Taiwanese, have a lot of mail in "big5",
> a charset commonly used for traditional Chinese, and some in UTF-8 or
> other DBCS (double-byte character sets for East Asian languages). Many
> of them couldn't be imported properly. I managed to solve most of the
> issues (I'll contribute to the ZoeDoc wiki if it's of value to others),
> but I need help with two problems.

Yes, if you could highlight the problems you went through, this would  
would be very valuable for me as well.

> The first is about 8bit characters in headers. When Zoe receives a
> message with 8bit headers and content (big5 characters sent verbatim,
> not encoded), the content is displayed correctly (if the "charset" is
> properly set), but the headers ("From" and/or "Subject" fields in
> Chinese) are not.
> I know 8bit headers are against RFC rules (right?),

Yes. Email headers should always be US-ASCII. Nothing else is allowed.  
If you use anything else, you need to encode it properly (RFC 2047).  
For example, the following subject in Korean:

(광고)당신을 너무너무 사랑했어요 나의 사랑을 받아...@ @

Is encoded as the following:

Subject:
   
=?euc-kr?q? 
(=B1=A4=B0=ED)=B4=E7=BD=C5=C0=BB_=B3=CA=B9=AB=B3=CA=B9=AB_=BB=E7=B6=FB=C 
7=DF=BE=EE=BF=E4_=B3=AA=C0=C7_=BB=E7=B6=FB=C0=BB_=B9=DE=BE=C6...@_@?=

>  but they're very
> commonly used in DBCS world.

Bummer :/

>  Since the content part is displayed fine, I
> believe ZOE does know how to handle such characters properly. The
> problem is (my guess), ZOE assumes the headers use only 7bit  
> characters.

Correct.

> It would be nice if ZOE can apply the information of "charset" to the
> headers, if it sees 8bit characters there. In other words, please treat
> 8bit characters in headers the same way the content is treated. That's
> what all the email clients I've tried (Becky, Thunderbird, The Bat!,  
> and
> Outlook Express) do.

Hmmm... I see... we can give it a try... could you send me a sample  
email which exhibit this behavior? If you have one of those in ZOE,  
click on the '@' (at) sign on the upper left side of a message and  
check "Attach Original Message":

http://zoe.nu/contents/Attach.jpg

Alternatively, Mozilla can forward emails as proper message/rfc822  
attachements as well.

> This is important for 8bit headers are still in use today by a lot of
> people. When importing such messages, my workaround is to redirect them
> using Becky, which is the only mail client I know that would re-encode
> headers when redirecting mail. All the others would simply add those
> "resend" headers and leave the original message unchanged.
>
> The workaround is time consuming, for Becky can redirect messages only  
> one
> at a time, asking for a recipient each time, but that's not my biggest
> problem. With new mail, Zoe retrieves a message often before I have a
> chance to do this trick. With the "delete duplicates" option enabled,
> the redirected message won't get in before I delete and purge the
> original one. (Can't turn off the option, for ZOE archives both my
> wife's and my mail, and we do have quite a few duplicates.)
>
> Enough for the first one. The second problem is with html-only mail in
> Chinese that uses 8bit content transfer encoding. With such mail, even
> when the charset is correct, the content isn't properly displayed. ZOE
> has no problem displaying multipart/mixed messages with both a plain
> text part and an html part. But with a pure html message, it generates  
> a
> text part without taking into account the charset setting (again, just  
> a
> guess).

Hmmm... this sounds like a bug... ZOE uses a third party library  
(JTidy) to handle the HTML parsing... I need to double check how the  
charset handling is done there... in any case, could you forward me a  
sample of such a message? This will help in tracking down the problem.

>
> My workaround was to export such mail to a text file, and manually
> created a plain text part by cleaning up the html codes in a text  
> editor.
> Reimport it into my mail client, and then redirect or forward it. The
> trick not only is time-consuming, but also fails with new mail, for the
> same reason stated above.
>
> Again, such mail is common place today, so a long-term solution is
> needed. I'm not a Java coder, but am otherwise willing to provide any
> help I can.

Ok... I think we can sort this out by adding some kind of additional  
heuristic while decoding the email headers to take into account non  
ASCII characters... regarding the HTML content, this is most likely a  
bug in the current parser which should be fix as well...

What would be very helpful though, is if you could provide a set of  
original emails which exhibit those problems so I can test whatever  
fixes need to be done on my side.

> Finally, thanks again for such great software. Despite all the  
> troubles,
> I think the effort is worthwhile -- if Zoe can handle new mail  
> properly.

Lets try to fix this :)

Thanks for reporting the problem!

Cheers,

R.



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl