$tt->error drops utf8 flag

--===============8688469486668373709==
Content-Type: multipart/alternative; boundary=90e6ba1efd8222518e051479b5a0

--90e6ba1efd8222518e051479b5a0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Template Toolkit version 2.23

This is a bit obscure, but the problem is TT is somehow turning off the
utf8 flag on $tt->error.

This can be a problem if trying to match text against the error returned
or, as in my case, sending the error to Log4perl, which (when configured)
will encode to utf8 as it should.

I'm wondering where and why TT is dropping the utf8 flag on the error
message.   It seems to happen when executing the complied template block.


Below I have simple code to demonstrate.

If the template generates an exception with wide-*character* data (e.g.
utf8 flag is on) the $tt->error string still comes back with the utf8 flag
OFF.

The example code below will generate an error like this -- note that the
utf8 flag is OFF:

Template ERROR [undef error - Chars are [=D0=A1=D0=B0=D1=80=D0=BB=D0=B0=D0=
=B3 =D1=85=D1=83=D1=81=D1=83=D1=83=D0=BB=D0=B0=D1=85 =D1=86=D0=B0=D0=B3 - =
=E1=80=91=E1=80=AD=E1=80=AF yak
=E1=80=9B=E1=80=AD=E1=80=90=E1=80=BA=E1=80=96=E1=80=AD=E1=80=AF=E1=80=B7=E1=
=80=A1=E1=80=81=E1=80=BB=E1=80=AD=E1=80=94=E1=80=BA]
] flag is [OFF]


then sending that to log4perl will essentially run encode_utf8() on the
error message and corrupt it like this, which is what I'm seeing in my logs=
.


undef error - Chars are [=C3=90=C2=A1=C3=90=C2=B0=C3=91=E2=82=AC=C3=90=C2=
=BB=C3=90=C2=B0=C3=90=C2=B3 =C3=91=E2=80=A6=C3=91=C6=92=C3=91 =C3=91=C6=92=
=C3=91=C6=92=C3=90=C2=BB=C3=90=C2=B0=C3=91=E2=80=A6 =C3=91=E2=80=A0=C3=90=
=C2=B0=C3=90=C2=B3 - =C3=A1=E2=82=AC=E2=80=98=C3=A1=E2=82=AC=C2=AD=C3=A1=E2=
=82=AC=C2=AF
yak =C3=A1=E2=82=AC=E2=80=BA=C3=A1=E2=82=AC=C2=AD=C3=A1=E2=82=AC =C3=A1=E2=
=82=AC=C2=BA=C3=A1=E2=82=AC=E2=80=93=C3=A1=E2=82=AC=C2=AD=C3=A1=E2=82=AC=C2=
=AF=C3=A1=E2=82=AC=C2=B7=C3=A1=E2=82=AC=C2=A1=C3=A1=E2=82=AC =C3=A1=E2=82=
=AC=C2=BB=C3=A1=E2=82=AC=C2=AD=C3=A1=E2=82=AC=E2=80=9D=C3=A1=E2=82=AC=C2=BA=
]



Now, if I remove or comment out the "diesub" and let the template process
normally then I get the template output back with the utf8 flag set, like
one would expect.

Template output [This is my template =D0=A1=D0=B0=D1=80=D0=BB=D0=B0=D0=B3 =
=D1=85=D1=83=D1=81=D1=83=D1=83=D0=BB=D0=B0=D1=85 =D1=86=D0=B0=D0=B3 - =E1=
=80=91=E1=80=AD=E1=80=AF yak
=E1=80=9B=E1=80=AD=E1=80=90=E1=80=BA=E1=80=96=E1=80=AD=E1=80=AF=E1=80=B7=E1=
=80=A1=E1=80=81=E1=80=BB=E1=80=AD=E1=80=94=E1=80=BA.
] flag is [ON]


So, it appears that when the complied template block is executed and an
exception happens the utf8 flag is getting cleared on $tt->error.






use strict;
use warnings;
use Template;
use Encode;

my $octets =3D '=D0=A1=D0=B0=D1=80=D0=BB=D0=B0=D0=B3 =D1=85=D1=83=D1=81=D1=
=83=D1=83=D0=BB=D0=B0=D1=85 =D1=86=D0=B0=D0=B3 - =E1=80=91=E1=80=AD=E1=80=
=AF yak =E1=80=9B=E1=80=AD=E1=80=90=E1=80=BA=E1=80=96=E1=80=AD=E1=80=AF=E1=
=80=A1=E1=80=81=E1=80=BB=E1=80=AD=E1=80=94=E1=80=BA';
my $char =3D decode_utf8( $octets );

my $tt =3D Template->new;


my $output;

$tt->process(
    \*DATA,
    {
        hello =3D> $char,
        diesub =3D> sub { die "Chars are [$char]\n" },
    },
    \$output
) || do {
    my $output =3D $tt->error;
    die printf "Template ERROR [%s] flag is [%s]\n",
        Encode::is_utf8( $output )
            ? ( encode_utf8( $output), 'ON' )
            : ($output, 'OFF');
};

printf "Template output [%s] flag is [%s]\n",
    Encode::is_utf8( $output )
        ? ( encode_utf8( $output), 'ON' )
        : ($output, 'OFF');


__END__
This is my template [% hello %].
[% diesub() %]



--=20
Bill Moseley
[email protected]

--90e6ba1efd8222518e051479b5a0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Template Toolkit version 2.23<br></div><div><br></div=
>This is a bit obscure, but the problem is TT is somehow turning off the ut=
f8 flag on $tt-&gt;error. =C2=A0=C2=A0<div><br></div><div>This can be a pro=
blem if trying to match text against the error returned or, as in my case, =
sending the error to Log4perl, which (when configured) will encode to utf8 =
as it should.<br><div><br></div><div><div>I&#39;m wondering where and why T=
T is dropping the utf8 flag on the error message. =C2=A0 It seems to happen=
 when executing the complied template block.</div><div><br></div><div><br><=
/div><div>Below I have simple code to demonstrate.</div><div><div><br></div=
><div>If the template generates an exception with wide-<i>character</i>=C2=
=A0data (e.g. utf8 flag is on) the $tt-&gt;error string still comes back wi=
th the=C2=A0utf8 flag OFF.</div><div><br></div><div><div>The example code b=
elow will generate an error like this -- note that the utf8 flag is OFF:</d=
iv><div><br></div><blockquote style=3D"margin:0px 0px 0px 40px;border:none;=
padding:0px"><div><div>Template ERROR [undef error - Chars are [=D0=A1=D0=
=B0=D1=80=D0=BB=D0=B0=D0=B3 =D1=85=D1=83=D1=81=D1=83=D1=83=D0=BB=D0=B0=D1=
=85 =D1=86=D0=B0=D0=B3 - =E1=80=91=E1=80=AD=E1=80=AF yak =E1=80=9B=E1=80=AD=
=E1=80=90=E1=80=BA=E1=80=96=E1=80=AD=E1=80=AF=E1=80=B7=E1=80=A1=E1=80=81=E1=
=80=BB=E1=80=AD=E1=80=94=E1=80=BA]</div></div><div><div>] flag is [OFF]</di=
v></div></blockquote><div><br></div><div>then sending that to log4perl will=
 essentially run encode_utf8() on the error message and corrupt it like thi=
s, which is what I&#39;m seeing in my logs.<br></div><blockquote style=3D"m=
argin:0px 0px 0px 40px;border:none;padding:0px"><blockquote style=3D"margin=
:0px 0px 0px 40px;border:none;padding:0px"><div><div><br></div></div></bloc=
kquote></blockquote><blockquote style=3D"margin:0px 0px 0px 40px;border:non=
e;padding:0px">undef error - Chars are [=C3=90=C2=A1=C3=90=C2=B0=C3=91=E2=
=82=AC=C3=90=C2=BB=C3=90=C2=B0=C3=90=C2=B3 =C3=91=E2=80=A6=C3=91=C6=92=C3=
=91 =C3=91=C6=92=C3=91=C6=92=C3=90=C2=BB=C3=90=C2=B0=C3=91=E2=80=A6 =C3=91=
=E2=80=A0=C3=90=C2=B0=C3=90=C2=B3 - =C3=A1=E2=82=AC=E2=80=98=C3=A1=E2=82=AC=
=C2=AD=C3=A1=E2=82=AC=C2=AF yak =C3=A1=E2=82=AC=E2=80=BA=C3=A1=E2=82=AC=C2=
=AD=C3=A1=E2=82=AC =C3=A1=E2=82=AC=C2=BA=C3=A1=E2=82=AC=E2=80=93=C3=A1=E2=
=82=AC=C2=AD=C3=A1=E2=82=AC=C2=AF=C3=A1=E2=82=AC=C2=B7=C3=A1=E2=82=AC=C2=A1=
=C3=A1=E2=82=AC =C3=A1=E2=82=AC=C2=BB=C3=A1=E2=82=AC=C2=AD=C3=A1=E2=82=AC=
=E2=80=9D=C3=A1=E2=82=AC=C2=BA]<br></blockquote><div><br></div><div><br></d=
iv><div>Now, if I remove or comment out the &quot;diesub&quot; and let the =
template process normally then I get the template output=C2=A0back with the=
 utf8 flag set, like one would expect.</div><div><br></div><blockquote styl=
e=3D"margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>Template ou=
tput [This is my template =D0=A1=D0=B0=D1=80=D0=BB=D0=B0=D0=B3 =D1=85=D1=83=
=D1=81=D1=83=D1=83=D0=BB=D0=B0=D1=85 =D1=86=D0=B0=D0=B3 - =E1=80=91=E1=80=
=AD=E1=80=AF yak =E1=80=9B=E1=80=AD=E1=80=90=E1=80=BA=E1=80=96=E1=80=AD=E1=
=80=AF=E1=80=B7=E1=80=A1=E1=80=81=E1=80=BB=E1=80=AD=E1=80=94=E1=80=BA.</div=
></div><div><div>] flag is [ON]</div></div></blockquote><div><br></div><div=
>So, it appears that when the complied template block is executed and an ex=
ception happens the utf8 flag is getting cleared on $tt-&gt;error.</div><di=
v><br></div><div><br></div><div><br></div><div><br></div><div><br></div><di=
v><br></div><div><div>use strict;</div><div>use warnings;</div><div>use Tem=
plate;</div><div>use Encode;</div><div><br></div><div>my $octets =3D &#39;=
=D0=A1=D0=B0=D1=80=D0=BB=D0=B0=D0=B3 =D1=85=D1=83=D1=81=D1=83=D1=83=D0=BB=
=D0=B0=D1=85 =D1=86=D0=B0=D0=B3 - =E1=80=91=E1=80=AD=E1=80=AF yak =E1=80=9B=
=E1=80=AD=E1=80=90=E1=80=BA=E1=80=96=E1=80=AD=E1=80=AF=E1=80=A1=E1=80=81=E1=
=80=BB=E1=80=AD=E1=80=94=E1=80=BA&#39;;</div><div>my $char =3D decode_utf8(=
 $octets );</div><div><br></div><div>my $tt =3D Template-&gt;new;</div><div=
><br></div><div><br></div><div>my $output;</div><div><br></div><div>$tt-&gt=
;process(</div><div>=C2=A0 =C2=A0 \*DATA,</div><div>=C2=A0 =C2=A0 {</div><d=
iv>=C2=A0 =C2=A0 =C2=A0 =C2=A0 hello =3D&gt; $char,</div><div>=C2=A0 =C2=A0=
 =C2=A0 =C2=A0 diesub =3D&gt; sub { die &quot;Chars are [$char]\n&quot; },<=
/div><div>=C2=A0 =C2=A0 },</div><div>=C2=A0 =C2=A0 \$output</div><div>) || =
do {</div><div>=C2=A0 =C2=A0 my $output =3D $tt-&gt;error;</div><div>=C2=A0=
 =C2=A0 die printf &quot;Template ERROR [%s] flag is [%s]\n&quot;,</div><di=
v>=C2=A0 =C2=A0 =C2=A0 =C2=A0 Encode::is_utf8( $output )</div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ? ( encode_utf8( $output), &#39;ON&#39; =
)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : ($output, &#39;OFF&=
#39;);</div><div>};</div><div><br></div><div>printf &quot;Template output [=
%s] flag is [%s]\n&quot;,</div><div>=C2=A0 =C2=A0 Encode::is_utf8( $output =
)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 ? ( encode_utf8( $output), &#39;ON&=
#39; )</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 : ($output, &#39;OFF&#39;);</d=
iv><div><br></div><div><br></div><div>__END__</div><div>This is my template=
 [% hello %].</div><div>[% diesub() %]</div><div><br></div><div><br clear=
=3D"all"><div><br></div>-- <br><div>Bill Moseley<br><a href=3D"mailto:mosel=
[email protected]" target=3D"_blank">[email protected]</a></div>
</div></div></div></div></div></div></div>

--90e6ba1efd8222518e051479b5a0--


--===============8688469486668373709==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
templates mailing list
[email protected]
http://mail.template-toolkit.org/mailman/listinfo/templates

--===============8688469486668373709==--