[ php-blog-Bugs-1461754 ] Permalinks broken for some UTF8 characters

Bugs item #1461754, was opened at 2006-03-31 01:57
Message generated for change (Comment added) made by cynebeald
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=542822&aid=1461754&group_id=75065

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Frontend functions
Group: CVS/SVN/Nightly
Status: Closed
Resolution: Fixed
Priority: 1
Submitted By: Kim Sullivan (cynebeald)
Assigned to: Garvin Hicking (garvinhicking)
Summary: Permalinks broken for some UTF8 characters

Initial Comment:
v1.0-beta2 (also present in the trunk on BerliOS)

The serendipity_makeFilename() function in
include/functions_permalinks.inc.php tries to apply the
utf8_decode() function to ALL utf-8 strings.

Unfortunately, when a character doesn't have a
ISO-8859-1 representation this function replaces it
with a ? (question mark), regardless of the original
character (it doesn't leave it intact).

This makes it impossible to use the $i18n_filename_from
and $i18n_filename_to variables for conversion (because
they are applied only after all "interesting"
characters have been converted to question marks).

This bug affects all locales that contain characters
not present in the ISO-8859-1 character set. An example
sentence is "Žluťoučký kůň příšerně úpěl ďábelské ódy"
from the czech locale which comes out of the
utf_decode() function as "?lu?ou?ký k?? p?í?ern? úp?l
?ábelskă ódy".

----------------------------------------------------------------------

>Comment By: Kim Sullivan (cynebeald)
Date: 2006-03-31 13:00

Message:
Logged In: YES 
user_id=1378043

Yes, thanks, it works now!

But the code looks a little weird, shouldn't the "default"
transliteration str_replace($from, $to, $str); be called in
all cases (since you call utf8-decode too...) From the
comments, I got the impression that the $i18n variables only
have to define some transliterations, and the rest should be
handled with the default transliteration. Something like this:

    if (isset($GLOBALS['i18n_filename_from']) &&
isset($GLOBALS['i18n_filename_to'])) {
        // Replace international chars detected by the
current language.
        // The array of chars is defined in the language file.
        $str = str_replace($GLOBALS['i18n_filename_from'],
$GLOBALS['i18n_filename_to'], $str);
    }
    
    //Replace common national chars from ISO-8859-1 not
detected by language file
    if (LANG_CHARSET == 'UTF-8') {
        // Try to convert all remaining unicode characters
to ISO-8859-1
        // Unknown characters will be replaced by '?'
        $str = utf8_decode($str);
    }
    //Apply default character transliterations
    $str = str_replace($from, $to, $str);

(this allows languages to use the default ' ' -> '-'
transliteration without explicitly defining it, while still
allowing overrides like in the russian locale where the
underscore is used for some reason).

----------------------------------------------------------------------

Comment By: Garvin Hicking (garvinhicking)
Date: 2006-03-31 10:31

Message:
Logged In: YES 
user_id=473563

Hi!

Thanks a lot for bringing this to attention. I have
committed this fix:
http://svn.berlios.de/viewcvs/serendipity?rev=1044&view=rev

This will do the utf8-transcoding AFTER the $i18n
transliteration. I think this should fix your problem? If
not, please report back!

Best regards,
Garvin

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=542822&aid=1461754&group_id=75065


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642