Re: Template Toolkit and UTF-8 template files

--===============7169558184677383140==
Content-Type: multipart/alternative; boundary=001a11c2c1fec8bdf20505eb8c11

--001a11c2c1fec8bdf20505eb8c11
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Did you use both ENCODING =3D> 'utf8' in "new" and binmode =3D>
':encoding(utf8)' in "process"?

There is more about it at this web page:

http://www.lemoda.net/perl/template-encoding/index.html




On 21 October 2014 16:44, David Hickman <[email protected]> wrote:

>  Dear all,
>
>
>
> I wondered if anyone had any experience of using the Perl Template Toolki=
t
> and UTF-8 encoded files? I appear to be facing a rather Strange issue wit=
h
> a UTF-8 encoded template file (although the only UTF-8 encoded characters
> in the file are the =C2=A3 sign):
>
>
>
> =E2=80=A2 If I don=E2=80=99t tell the template toolkit anything about the=
 encoding of the
> template file then the resulting output from the template is correct and
> the =C2=A3 signs are output in UTF-8 format
>
>
>
> =E2=80=A2 If I specifically tell the template toolkit that the file is en=
coded in
> UTF-8 (either using the configuration parameters, a BOM on the template
> file, or both) then the =C2=A3 signs in the template are converted to cha=
racter
> code 163 (the ANSI equivalent). This breaks my intended output of the
> template as the character encoding reported to the browser in the respons=
e
> header is UTF-8 even though the file now contains characters that are  no=
t
> compatible with UTF-8.
>
>
>
> =E2=80=A2 If I add some =E2=80=9Cdefinitely Unicode=E2=80=9D characters (=
=E8=87=AA=E6=B2=B3=E5=8D=97=E7=BB=8F=E4=B9=B1=E5=85=B3) to the template
> and don=E2=80=99t change anything else in the file at all, then the outpu=
t from the
> file works fine regardless of whether I tell the template toolkit that th=
e
> file is UTF-8 or not.
>
>
>
>
>
> Does anyone have any ideas about what might be going on and how I could
> fix this issue?  From what I have found from my investigation it appears =
as
> though the =C2=A3 sign sits in a character range that can cause Perl to c=
hange
> its interpretation of a string and I wonder if this might be causing a
> problem somewhere down the line?  It might be worth mentioning that the
> template toolkit is configured to compile templates in memory rather than
> writing to disk.
>
>
>
>
>
> Any pointers would be most gratefully received as I am rather stuck!
>
>
>
>
>
> Best Regards,
>
> David
>
>
> _______________________________________________
> templates mailing list
> [email protected]
> http://mail.template-toolkit.org/mailman/listinfo/templates
>
>

--001a11c2c1fec8bdf20505eb8c11
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Did you use both ENCODING =3D&gt; &#39;utf8&#39; in &quot;=
new&quot; and binmode =3D&gt; &#39;:encoding(utf8)&#39; in &quot;process&qu=
ot;?<div><br></div><div>There is more about it at this web page:</div><div>=
<br></div><div><a href=3D"http://www.lemoda.net/perl/template-encoding/inde=
x.html">http://www.lemoda.net/perl/template-encoding/index.html</a><br></di=
v><div><br></div><div><br></div><div><br></div></div><div class=3D"gmail_ex=
tra"><br><div class=3D"gmail_quote">On 21 October 2014 16:44, David Hickman=
 <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]" target=
=3D"_blank">[email protected]</a>&gt;</span> wrote:<br><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid=
;padding-left:1ex">





<div lang=3D"EN-GB" link=3D"blue" vlink=3D"purple">
<div>
<p class=3D"MsoNormal">Dear all,<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">I wondered if anyone had any experience of using the=
 Perl Template Toolkit and UTF-8 encoded files? I appear to be facing a rat=
her Strange issue with a UTF-8 encoded template file (although the only UTF=
-8 encoded characters in the file
 are the =C2=A3 sign):<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">=E2=80=A2 If I don=E2=80=99t tell the template toolk=
it anything about the encoding of the template file then the resulting outp=
ut from the template is correct and the =C2=A3 signs are output in UTF-8 fo=
rmat<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">=E2=80=A2 If I specifically tell the template toolki=
t that the file is encoded in UTF-8 (either using the configuration paramet=
ers, a BOM on the template file, or both) then the =C2=A3 signs in the temp=
late are converted to character code 163 (the ANSI
 equivalent). This breaks my intended output of the template as the charact=
er encoding reported to the browser in the response header is UTF-8 even th=
ough the file now contains characters that are =C2=A0not compatible with UT=
F-8.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">=E2=80=A2 If I add some =E2=80=9Cdefinitely Unicode=
=E2=80=9D characters (<span lang=3D"EN-US" style=3D"font-family:&quot;MS Go=
thic&quot;">=E8=87=AA=E6=B2=B3=E5=8D=97</span><span lang=3D"EN-US" style=3D=
"font-family:MingLiU">=E7=BB=8F=E4=B9=B1=E5=85=B3</span>) to the template a=
nd don=E2=80=99t change anything else in the file at all,
 then the output from the file works fine regardless of whether I tell the =
template toolkit that the file is UTF-8 or not.<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Does anyone have any ideas about what might be going=
 on and how I could fix this issue?=C2=A0 From what I have found from my in=
vestigation it appears as though the =C2=A3 sign sits in a character range =
that can cause Perl to change its interpretation
 of a string and I wonder if this might be causing a problem somewhere down=
 the line?=C2=A0 It might be worth mentioning that the template toolkit is =
configured to compile templates in memory rather than writing to disk.<u></=
u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Any pointers would be most gratefully received as I =
am rather stuck!<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p>
<p class=3D"MsoNormal">Best Regards,<u></u><u></u></p>
<p class=3D"MsoNormal">David<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u><u></u></p>
</div>
</div>

<br>_______________________________________________<br>
templates mailing list<br>
<a href=3D"mailto:[email protected]">templates@template-toolki=
t.org</a><br>
<a href=3D"http://mail.template-toolkit.org/mailman/listinfo/templates" tar=
get=3D"_blank">http://mail.template-toolkit.org/mailman/listinfo/templates<=
/a><br>
<br></blockquote></div><br></div>

--001a11c2c1fec8bdf20505eb8c11--


--===============7169558184677383140==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
templates mailing list
[email protected]
http://mail.template-toolkit.org/mailman/listinfo/templates

--===============7169558184677383140==--