Cannot decode string with wide characters

[email protected] (Bill Moseley) Wed, 4 Dec 2013 21:10:02 -0800
Newsgroups perl.unicode
Message-ID <CAKhN_m6dLTZBs-j40S3C_GXMvXaU1BJ3Xn2xy22L+=kWpaYkdw@mail.gmail.com>
--047d7bacb0b49300da04ecc28c9a
Content-Type: text/plain; charset=ISO-8859-1

"Cannot decode string with wide characters"

I see this error when calling decode_utf8() on a character string (with
wide characters) that was already decoded.

This error started with Encode 2.53, it seems.  Previous versions appear to
ignore the double decoding.   I assume that means the decode was a noop if
the UTF8 flag was already on.

I'm curious about the change.  Was there a problem with the old behavior,
or was the change to make it more strict?   Or some other problem caused by
double decoding?

I agree that it would be good to be told when double-decoding.

But, I have a situation with a legacy app where decode_utf8 was sprinkled
all over the app because the database DBD was not configured to turn on the
UTF8 flag (pg_enable_utf8).   Turning it on now would be the right approach
and solve some existing encoding issues, but the newer Encode breaks
because of all the existing decode_utf8 calls.


The joys of legacy apps:

$ fgrep  -r 'decode_utf8(' . | wc -l
456

-- 
Bill Moseley
[email protected]

--047d7bacb0b49300da04ecc28c9a
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>&quot;Cannot decode string with wide characters&quot;=
</div><div><br></div><div>I see this error when calling decode_utf8() on a =
character string (with wide characters) that was already decoded.</div><div=
>

<br></div>This error started with Encode 2.53, it seems. =A0Previous versio=
ns appear to ignore the double decoding. =A0 I assume that means the decode=
 was a noop if the UTF8 flag was already on.<div>
<br></div><div>I&#39;m curious about the change. =A0Was there a problem wit=
h the old behavior, or was the change to make it more strict? =A0 Or some o=
ther problem caused by double decoding?</div><div><br></div><div>I agree th=
at it would be good to be told when double-decoding.</div>

<div><br></div><div>But, I have a situation with a legacy app where decode_=
utf8 was sprinkled all over the app because the database DBD was not config=
ured to turn on the UTF8 flag (pg_enable_utf8). =A0 Turning it on now would=
 be the right approach and solve some existing encoding issues, but the new=
er Encode breaks because of all the existing decode_utf8 calls.</div>

<div><br></div><div><br></div><div>The joys of legacy apps:</div><div><br><=
/div><div><div><div><div>$ fgrep =A0-r &#39;decode_utf8(&#39; . | wc -l</di=
v><div>456</div></div><div><div><br></div>-- <br>Bill Moseley<br><a href=3D=
"mailto:[email protected]" target=3D"_blank">[email protected]</a>
</div></div></div></div>

--047d7bacb0b49300da04ecc28c9a--