Re: DBD::Pg && handling of UTF-8 char columns

--TB36FDmn/VVEgNH/
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

El d=C3=ADa viernes, octubre 11, 2019 a las 04:03:31p. m. -0600, Jon Jensen=
 escribi=C3=B3:

> Perl's internal storage of string data is a little odd. \xe4 is the=20
> correct Unicode code point as per:
>=20
> https://en.wikipedia.org/wiki/Latin-1_Supplement_%28Unicode_block%29
>=20
> It is not UTF-8 encoded, true, but there's no reason Perl internally need=
s=20
> to use UTF-8 specifically, and I believe for Latin-1 it does not by=20
> default. It's a question of in-memory storage and processing (some kind o=
f=20
> Unicode) vs. input/output (where you want UTF-8).
>=20
> If your script is configured to send UTF-8 to STDOUT, then I would expect=
=20
> that \xe4 will show up as the UTF-8 \xc3\xa4 instead.

I inserted another row into this table, encoded in UTF-8:

pos71=3D# select d02name from d02ben where d02bnr =3D '08.05.1945' ;
 =D0=BE=D1=81=D0=B2=D0=BE=D0=B1=D0=BE=D0=B6=D0=B4=D0=B5=D0=BD=D0=B8=D0=B5

pos71=3D# select d02name::bytea from d02ben where d02bnr =3D '08.05.1945' ;
 \xd0bed181d0b2d0bed0b1d0bed0b6d0b4d0b5d0bdd0b8d0b520202020202020 ...

If I run this through Perl DBD::Pg:

   @row =3D $sth->fetchrow_array;
   $HexStr =3D unpack("H*", $row[0]);
   print "HexStr: " . $HexStr . "\n";
   print "$row[0]\n";

   binmode(STDOUT, ':encoding(utf8)');
   print "after binmode: $row[0]\n";


it gives:

DBI is version 1.642, DBD::Pg is version 3.10.0
client_encoding=3DUTF8, server_encoding=3DUTF8
HexStr: 3e41323e313e3634353d38352020202020202020 ...
Wide character in print at ./utf8-01.pl line 66.
=D0=BE=D1=81=D0=B2=D0=BE=D0=B1=D0=BE=D0=B6=D0=B4=D0=B5=D0=BD=D0=B8=D0=B5
after binmode: =D0=BE=D1=81=D0=B2=D0=BE=D0=B1=D0=BE=D0=B6=D0=B4=D0=B5=D0=BD=
=D0=B8=D0=B5

and if I add an utf8::encode($row[0]) after the fetch, like:

   @row =3D $sth->fetchrow_array;
   utf8::encode($row[0]);

it gives the correkt UTF-8 encoding:

DBI is version 1.642, DBD::Pg is version 3.10.0
client_encoding=3DUTF8, server_encoding=3DUTF8
HexStr: d0bed181d0b2d0bed0b1d0bed0b6d0b4d0b5d0bdd0b8d0b520202020202020 ...
=D0=BE=D1=81=D0=B2=D0=BE=D0=B1=D0=BE=D0=B6=D0=B4=D0=B5=D0=BD=D0=B8=D0=B5
after binmode: =C3=90=C2=BE=C3=91=C3=90=C2=B2=C3=90=C2=BE=C3=90=C2=B1=C3=90=
=C2=BE=C3=90=C2=B6=C3=90=C2=B4=C3=90=C2=B5=C3=90=C2=BD=C3=90=C2=B8=C3=90=C2=
=B5

i.e. the array returned by $sth->fetchrow_array does not contain an UTF-8 s=
tring.

Why it has to be passed through utf8::encode($row[0]) ?

Thanks

	matthias

--=20
Matthias Apitz, =E2=9C=89 [email protected], http://www.unixarea.de/ +49-176=
-38902045
Public GnuPG key: http://www.unixarea.de/key.pub

3. Oktober! Wir gratulieren! Der Berliner Fernsehturm wird 50=20
aus: https://www.jungewelt.de/2019/10-02/index.php

--TB36FDmn/VVEgNH/
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEXmn7rBYYViyzy/vBR8z35Hb+nREFAl2hj24ACgkQR8z35Hb+
nREGEQ/+OZgBfE3MfH5xiwh4hWaqS43gS7rhEL8MloW4c00/GTvWhh9rMmo84c17
bJjdGwKiFQRdOOcT+9lWW35aMyqJST755r36r5r3+N0SHxHCEu2R3EzONIqPwl/P
zi+5qs18sQp6zWJN4zHeaIFEPFVJREOZGG8HKr76icwWFuT7uR3mH0Ysf9Q6YEG+
QjOtt0XqSIAypYVYp8IcW+dWqc0il09ZePZB0KnQSoUED246uHHD4D9PhyqukUfR
BgrYjEsIqNPA9tJTow1StrDRcSwMqeyBYJ0y7vMQQ1T8F8SKz3kLP8XngJi4N9Pw
kUubmzbl8Q3sIArO0E5VabMM2dGGrHqp4XbQ5aJ5J+vVtXks6URm0BnXDJN1u027
8Yyf6/3nvWGuHIfvV2Epr24tqB+9tGU2iQ3gWpukJT415BL88PEIA2JQ9H0CF2fc
nTWimmCzrDClVEVaTayBTbHrRrIB+tIbxginsgr4ACvs7K8Nu5IH+K6SIxUmcndF
hdcYX23A6Q1IxHvRWuJisAz0B+Y6EIbejXyhfZdwODxMkUHgYaXoF7+P4btoMAiu
i5bd/bde3f8vJV+zpT7fp36HS0k+T39Za4rFmeDRwHznmapugRCTdDpULPJWbF3p
ESVtSFioL/w/RhYJwnmwt2EIC4PB4dMikJapUlsVcnuFwnOAlpE=
=RtNf
-----END PGP SIGNATURE-----

--TB36FDmn/VVEgNH/--