Re: need help with utf-8

--0000000000003c6d3406296a9570
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Dec 16, 2024 at 5:13=E2=80=AFPM Shaomei Liu <[email protected]=
om>
wrote:

> Hello,
> very happy to find this mailing list as it is my last resort!!
> I have a project which uses DBI to write to postgres DB.
> after upgrading from RHEL7 to RHEL8, the utf-8 character is not displayed
> properly in the DB. DB has correct utf-8 encoding set.
> for example, left double quotation mark   =E2=80=9C  is displayed as =C3=
=A2\u0080\u009C
> .
> You can use this link to check hex utf-8 bytes
> https://www.cogsci.ed.ac.uk/~richard/utf-8.cgi?input=3D%E2%80%9C&mode=3Dc=
har
>
> below is the file testutf.pl which writes left double quotation mark  =E2=
=80=9C
> to the database. it also shows the query results from psql for both EL8 a=
nd
> EL7.
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3Dfile testutf.pl=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
> #!/usr/bin/perl
> use strict;
> use warnings;
> use DBI;
> print "DBI version: $DBI::VERSION\n";
>
> my $db =3D "debugutf";
> my $host =3D "db";
> my $user =3D "postgres";
> my $pass =3D "";
> my $dbh =3D DBI->connect("DBI:Pg:dbname=3D$db;host=3D$host",$user,$pass);
> my $sql =3D 'INSERT INTO table1 (title) VALUES (?)';
> my $query =3D $dbh->prepare($sql);
> my @values =3D ('=E2=80=9C');
> $query->execute(@values);
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3Don RHEL8
> #execute testutf.pl which wrote =E2=80=9C to database on RHEL8
> text.tac1.dev.bia-boeing.com> ./testutf.pl
> DBI version: 1.641
>
> #from psql
> debugutf=3D# select * from table1;
>      title
> ---------------
>  =C3=A2\u0080\u009C  =3D=3D=3D=3D=3D=3D=3D=3D=3D>unexpected
> (1 row)
>
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3Don RHEL7
> #execute testutf.pl which wrote =E2=80=9C to database on RHEL8
> text.tac1.dev.bia-boeing.com> ./testutf.pl
> DBI version: 1.627
>
> #from psql
> debugutf=3D# select * from table1;
>      title
> ---------------
>  =E2=80=9C       =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>expected
> (1 row)
>
> Any feedback is appreciated.
> thank you
> Shirley
>

Hello,

This is most likely due to changes in the version of DBD::Pg you are using.
Make sure you include the declaration "use utf8;" in a script where you
will write non-ascii literal strings in the source code, and ensure the
script is written in UTF-8 encoding (the default of most text editors these
days). If you are getting strings from elsewhere, you will need to ensure
that they are being decoded from the UTF-8 encoding in whatever way is
appropriate - for example, Mojolicious automatically decodes request
parameters from UTF-8, and using the ':encoding(UTF-8)' layer or read_text
from File::Slurper will decode text read from a UTF-8-encoded file.

-Dan

--0000000000003c6d3406296a9570
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">On Mon, Dec 16, 2024 at 5:13=E2=80=AFPM S=
haomei Liu &lt;<a href=3D"mailto:[email protected]">sliu.newjersey@g=
mail.com</a>&gt; wrote:</div><div class=3D"gmail_quote gmail_quote_containe=
r"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord=
er-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Hello=
,<br>very happy to find this mailing list as it is my last resort!!<div>I h=
ave a project which uses DBI to write to postgres DB.</div><div>after upgra=
ding from RHEL7 to RHEL8, the utf-8 character is not displayed properly in =
the DB. DB has correct utf-8 encoding set.</div><div>for example, left doub=
le quotation mark=C2=A0 =C2=A0=E2=80=9C=C2=A0 is displayed as=C2=A0<span st=
yle=3D"background-color:rgb(31,31,31);color:rgb(204,204,204);font-family:Co=
nsolas,&quot;Courier New&quot;,monospace;font-size:14px;white-space:pre-wra=
p">=C3=A2\u0080\u009C</span>.</div><div>You can use this link to check hex =
utf-8 bytes</div><div><a href=3D"https://www.cogsci.ed.ac.uk/~richard/utf-8=
.cgi?input=3D%E2%80%9C&amp;mode=3Dchar" target=3D"_blank">https://www.cogsc=
i.ed.ac.uk/~richard/utf-8.cgi?input=3D%E2%80%9C&amp;mode=3Dchar</a></div><d=
iv><br></div><div>below is the file <a href=3D"http://testutf.pl" target=3D=
"_blank">testutf.pl</a> which writes left double quotation mark=C2=A0

=E2=80=9C to the database. it also shows the query results from psql for bo=
th EL8 and EL7.<br><br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3Dfile <a href=3D"http:/=
/testutf.pl" target=3D"_blank">testutf.pl</a>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
<br>#!/usr/bin/perl<br>use strict;<br>use warnings;<br>use DBI;<br>print &q=
uot;DBI version: $DBI::VERSION\n&quot;;<br><br>my $db =3D &quot;debugutf&qu=
ot;;<br>my $host =3D &quot;db&quot;;<br>my $user =3D &quot;postgres&quot;;<=
br>my $pass =3D &quot;&quot;;<br>my $dbh =3D DBI-&gt;connect(&quot;DBI:Pg:d=
bname=3D$db;host=3D$host&quot;,$user,$pass);<br>my $sql =3D &#39;INSERT INT=
O table1 (title) VALUES (?)&#39;;<br>my $query =3D $dbh-&gt;prepare($sql);<=
br>my @values =3D (&#39;=E2=80=9C&#39;);<br>$query-&gt;execute(@values);<br=
>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br><br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3Don RHEL8<br>#execute <a href=3D"http://testutf.pl" target=3D"_blan=
k">testutf.pl</a> which wrote =E2=80=9C to database on RHEL8<br><a href=3D"=
http://text.tac1.dev.bia-boeing.com" target=3D"_blank">text.tac1.dev.bia-bo=
eing.com</a>&gt; ./<a href=3D"http://testutf.pl" target=3D"_blank">testutf.=
pl</a><br>DBI version: 1.641<br><br>#from psql<br>debugutf=3D# select * fro=
m table1;<br>=C2=A0 =C2=A0 =C2=A0title<br>---------------<br>=C2=A0<font co=
lor=3D"#ff0000">=C3=A2\u0080\u009C=C2=A0 =3D=3D=3D=3D=3D=3D=3D=3D=3D&gt;une=
xpected</font><br>(1 row)<br><br><br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3Don RHEL7<br>#execute <a href=3D"http://testutf.pl" target=3D"_blank">=
testutf.pl</a> which wrote =E2=80=9C to database on RHEL8<br><a href=3D"htt=
p://text.tac1.dev.bia-boeing.com" target=3D"_blank">text.tac1.dev.bia-boein=
g.com</a>&gt; ./<a href=3D"http://testutf.pl" target=3D"_blank">testutf.pl<=
/a><br>DBI version: 1.627<br><br>#from psql<br>debugutf=3D# select * from t=
able1;<br>=C2=A0 =C2=A0 =C2=A0title<br>---------------<br><font color=3D"#0=
000ff">=C2=A0=E2=80=9C=C2=A0 =C2=A0 =C2=A0 =C2=A0=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D&gt;expected</font><br>(1 row)</div><div><br></div><div>Any fee=
dback is appreciated.</div><div>thank you</div><div>Shirley</div></div></bl=
ockquote><div><br></div><div>Hello,</div><div><br></div><div>This is most l=
ikely due to changes in the version of DBD::Pg you are using. Make sure you=
 include the declaration &quot;use utf8;&quot; in a script where you will w=
rite non-ascii literal strings in the source code, and ensure the script is=
 written in UTF-8 encoding (the default of most text editors these days). I=
f you are getting strings from elsewhere, you will need to ensure that they=
 are being decoded from the UTF-8 encoding in whatever way is appropriate -=
 for example, Mojolicious automatically decodes request parameters from UTF=
-8, and using the &#39;:encoding(UTF-8)&#39; layer or read_text from File::=
Slurper will decode text read from a UTF-8-encoded file.</div><div><br></di=
v><div>-Dan</div></div></div>

--0000000000003c6d3406296a9570--