Re: need help with utf-8
[email protected] (Alexander Foken via dbi-users) Tue, 17 Dec 2024 21:30:08 +0100
| Newsgroups | perl.dbi.users |
|---|---|
| Message-ID | <[email protected]> |
--------------M7hLBj9UJ0ahZ4Cwp0KlCGcl Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi, DBD::ODBC has several tests related to Unicode handling (40UnicodeRoundTrip.t, 41Unicode.t, 45_unicode_varchar.t), they should also work with other DBDs. They should tell you if your problem is between Perl and Postgres or if it is simply in the encoding of your terminal. Alexander On 17.12.2024 13:31, Felipe Gasper via dbi-users wrote: > Respectfully to Dan & others, I don’t advocate adding “use utf8” to > existing code without a clear understanding of where your program’s > decode & encode points are. > > Check to see what DBD::Pg actually writes to the database. If it > suddenly started encoding, that’s a breaking change that either was > documented or should be reported upstream. > >> On Dec 16, 2024, at 17:13, Shaomei Liu <[email protected]> wrote: >> >> >> Hello, >> very happy to find this mailing list as it is my last resort!! >> I have a project which uses DBI to write to postgres DB. >> after upgrading from RHEL7 to RHEL8, the utf-8 character is not >> displayed properly in the DB. DB has correct utf-8 encoding set. >> for example, left double quotation mark “ is displayed as >> â\u0080\u009C. >> You can use this link to check hex utf-8 bytes >> https://www.cogsci.ed.ac.uk/~richard/utf-8.cgi?input=%E2%80%9C&mode=char >> <https://www.cogsci.ed.ac.uk/~richard/utf-8.cgi?input=%E2%80%9C&mode=char> >> >> below is the file testutf.pl <http://testutf.pl> which writes left >> double quotation mark “ to the database. it also shows the query >> results from psql for both EL8 and EL7. >> >> ==========file testutf.pl <http://testutf.pl>========== >> #!/usr/bin/perl >> use strict; >> use warnings; >> use DBI; >> print "DBI version: $DBI::VERSION\n"; >> >> my $db = "debugutf"; >> my $host = "db"; >> my $user = "postgres"; >> my $pass = ""; >> my $dbh = DBI->connect("DBI:Pg:dbname=$db;host=$host",$user,$pass); >> my $sql = 'INSERT INTO table1 (title) VALUES (?)'; >> my $query = $dbh->prepare($sql); >> my @values = ('“'); >> $query->execute(@values); >> =================================== >> >> ==============on RHEL8 >> #execute testutf.pl <http://testutf.pl> which wrote “ to database on >> RHEL8 >> text.tac1.dev.bia-boeing.com <http://text.tac1.dev.bia-boeing.com>> >> ./testutf.pl <http://testutf.pl> >> DBI version: 1.641 >> >> #from psql >> debugutf=# select * from table1; >> title >> --------------- >> â\u0080\u009C =========>unexpected >> (1 row) >> >> >> ==============on RHEL7 >> #execute testutf.pl <http://testutf.pl> which wrote “ to database on >> RHEL8 >> text.tac1.dev.bia-boeing.com <http://text.tac1.dev.bia-boeing.com>> >> ./testutf.pl <http://testutf.pl> >> DBI version: 1.627 >> >> #from psql >> debugutf=# select * from table1; >> title >> --------------- >> “ ============>expected >> (1 row) >> >> Any feedback is appreciated. >> thank you >> Shirley -- Alexander Foken mailto:[email protected] --------------M7hLBj9UJ0ahZ4Cwp0KlCGcl Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <p>Hi,</p> <p>DBD::ODBC has several tests related to Unicode handling (40UnicodeRoundTrip.t, 41Unicode.t, 45_unicode_varchar.t), they should also work with other DBDs. They should tell you if your problem is between Perl and Postgres or if it is simply in the encoding of your terminal.<br> </p> <p>Alexander<br> </p> <div class="moz-cite-prefix">On 17.12.2024 13:31, Felipe Gasper via dbi-users wrote:<br> </div> <blockquote type="cite" cite="mid:[email protected]"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div dir="ltr">Respectfully to Dan & others, I don’t advocate adding “use utf8” to existing code without a clear understanding of where your program’s decode & encode points are.</div> <div dir="ltr"><br> </div> <div dir="ltr">Check to see what DBD::Pg actually writes to the database. If it suddenly started encoding, that’s a breaking change that either was documented or should be reported upstream.</div> <div dir="ltr"><br> <blockquote type="cite">On Dec 16, 2024, at 17:13, Shaomei Liu <a class="moz-txt-link-rfc2396E" href="mailto:[email protected]"><[email protected]></a> wrote:<br> <br> </blockquote> </div> <blockquote type="cite"> <div dir="ltr"> <div dir="ltr">Hello,<br> very happy to find this mailing list as it is my last resort!! <div>I have a project which uses DBI to write to postgres DB.</div> <div>after upgrading from RHEL7 to RHEL8, the utf-8 character is not displayed properly in the DB. DB has correct utf-8 encoding set.</div> <div>for example, left double quotation mark “ is displayed as <span style="background-color:rgb(31,31,31);color:rgb(204,204,204);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre">â\u0080\u009C</span>.</div> <div>You can use this link to check hex utf-8 bytes</div> <div><a href="https://www.cogsci.ed.ac.uk/~richard/utf-8.cgi?input=%E2%80%9C&mode=char" moz-do-not-send="true">https://www.cogsci.ed.ac.uk/~richard/utf-8.cgi?input=%E2%80%9C&mode=char</a></div> <div><br> </div> <div>below is the file <a href="http://testutf.pl" moz-do-not-send="true">testutf.pl</a> which writes left double quotation mark “ to the database. it also shows the query results from psql for both EL8 and EL7.<br> <br> ==========file <a href="http://testutf.pl" moz-do-not-send="true">testutf.pl</a>==========<br> #!/usr/bin/perl<br> use strict;<br> use warnings;<br> use DBI;<br> print "DBI version: $DBI::VERSION\n";<br> <br> my $db = "debugutf";<br> my $host = "db";<br> my $user = "postgres";<br> my $pass = "";<br> my $dbh = DBI->connect("DBI:Pg:dbname=$db;host=$host",$user,$pass);<br> my $sql = 'INSERT INTO table1 (title) VALUES (?)';<br> my $query = $dbh->prepare($sql);<br> my @values = ('“');<br> $query->execute(@values);<br> ===================================<br> <br> ==============on RHEL8<br> #execute <a href="http://testutf.pl" moz-do-not-send="true">testutf.pl</a> which wrote “ to database on RHEL8<br> <a href="http://text.tac1.dev.bia-boeing.com" moz-do-not-send="true">text.tac1.dev.bia-boeing.com</a>> ./<a href="http://testutf.pl" moz-do-not-send="true">testutf.pl</a><br> DBI version: 1.641<br> <br> #from psql<br> debugutf=# select * from table1;<br> title<br> ---------------<br> <font color="#ff0000">â\u0080\u009C =========>unexpected</font><br> (1 row)<br> <br> <br> ==============on RHEL7<br> #execute <a href="http://testutf.pl" moz-do-not-send="true">testutf.pl</a> which wrote “ to database on RHEL8<br> <a href="http://text.tac1.dev.bia-boeing.com" moz-do-not-send="true">text.tac1.dev.bia-boeing.com</a>> ./<a href="http://testutf.pl" moz-do-not-send="true">testutf.pl</a><br> DBI version: 1.627<br> <br> #from psql<br> debugutf=# select * from table1;<br> title<br> ---------------<br> <font color="#0000ff"> “ ============>expected</font><br> (1 row)</div> <div><br> </div> <div>Any feedback is appreciated.</div> <div>thank you</div> <div>Shirley</div> </div> </div> </blockquote> </blockquote> <pre class="moz-signature" cols="72">-- Alexander Foken <a class="moz-txt-link-freetext" href="mailto:[email protected]">mailto:[email protected]</a></pre> </body> </html> --------------M7hLBj9UJ0ahZ4Cwp0KlCGcl--