[ php-blog-Bugs-2831116 ] POPfetcher Charset Conversion Badly Broken

Bugs item #2831116, was opened at 2009-08-02 14:19
Message generated for change (Comment added) made by btittelbach
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=542822&aid=2831116&group_id=75065

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Plugins
Group: None
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: B.Tittelbach (btittelbach)
Assigned to: Nobody/Anonymous (nobody)
Summary: POPfetcher Charset Conversion Badly Broken

Initial Comment:
Tested with Serendipity 1.4.1, downloaded newest POPfetcher Plugin with Spartacus.

Problem: Character Conversion of e-mails is not implemented correctly. Only works with latin1 <-> utf-8.
the php Functions utf8_en/decode($string) are used, which only de/encodes latin1 to/from utf8.

Instead, depending on avaible php extension, the following should be used:

in PHP6:
function decode($string, $charset) {
            return unicode_decode($string,$charset);
}

in PHP4/5 with iconv compiled in:
function decode($string, $charset) {
            $destcc=LANG_CHARSET;
            if (!empty($destcc)) $destcc="utf-8";
            return iconv($charset,$destcc,$string);   
}

in PHP4/5 with recodelib extension enabled:
function decode($string, $charset) {
            $destcc=LANG_CHARSET;
            if (!empty($destcc)) $destcc="utf-8";
            return recode_string("$charset..$destcc",$string);
}

utf8_en/decode can _only_ be used if latin1<->utf-8 conversion is needed. Otherwise an error should be thrown.

----------------------------------------------------------------------

>Comment By: B.Tittelbach (btittelbach)
Date: 2009-08-06 16:13

Message:
Thanks, works perfectly now :)
I think that resolves the issue, so I'm closing the bug.

----------------------------------------------------------------------

Comment By: B.Tittelbach (btittelbach)
Date: 2009-08-06 16:13

Message:
Based on your report we were able to solve this problem.

The problem has therefore been fixed in our CVS Repository and
will appear in our next release.

Thank you for helping us making serendipity better.

----------------------------------------------------------------------

Comment By: Garvin Hicking (garvinhicking)
Date: 2009-08-06 10:12

Message:
Hi!

Thanks for that information, I now see your point. I just committed a
slightly modified version of your patch, please tell me if that also works
out for you?

Best regards,
Garvin

----------------------------------------------------------------------

Comment By: B.Tittelbach (btittelbach)
Date: 2009-08-05 17:05

Message:
Serendipidy as well as my whole system (php, apache, etc included) is
UTF-8.
The Mail was encoded ISO-2022-JP. 
I found that decoding the junk that arrived in serendipity from UTF-8 to
latin1 yielded correct UTF-8.
So first is suspected that mime did the decoding correclty and that your
decode() function would incorreclty convert the stuff once again to latin1.
Adding a "return $string" in the first line of decode() however, just
yielded an ISO-2022-JP encoded entry in serendipity (which of course was
shown incorretly as serendipitdy announced the page as utf-8)

The problem was resolved when I replaced the content of the decode(..)
function with
....
return iconv($charset,$destcc,$string); 

That suggests to me that the mimeDecode Lib does not decode the mail or at
the very least does not decode mails in this encoding.

Greetings and thanks for writing PopFetcher in the first place :)

----------------------------------------------------------------------

Comment By: Garvin Hicking (garvinhicking)
Date: 2009-08-05 11:59

Message:
Hi!

Can you give a description of your setup where you faced the bug? Which
charset did you configure for serendipity, and in which charset was your
mail?

The mimeDecode library also translates the charsets, AFAIK - so what I
have in mind was that it takes care of always either supplying latin1 or
UTF-8 content. If that's not the case, I'd like to setup an installation
that matches your to investigate the issue properly.

Thanks,
Garvin

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=542822&aid=2831116&group_id=75065

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july