Re: floats truncated by strtod() if decimal point character differs
Michael Weiser <[email protected]> Tue, 27 Aug 2019 13:48:52 +0200
| Newsgroups | gmane.comp.db.postgresql.dbdpg |
|---|---|
| Message-ID | <[email protected]> |
Hello Noah,
On Mon, Aug 26, 2019 at 11:28:09PM -0700, Noah Misch wrote:
> > > I've run into a problem where floats loose their fractions on the way
> > > from the database to a perl script using DBD::Pg. A minimal reproduce=
r:
> > Am I wrong on this list with my question or was there just too much or
> > too little or too specific information at once? :)
> It's the right list. The issue was complex enough to require some
> uninterrupted study time.
Thanks for getting back to me!
> > > # LANG=3DC perl ~/t.pl
> > > FOO: 1.1:1.100000 C
> > > 1.1
> > > # LANG=3Dde_DE.UTF-8 perl ~/t.pl
> > > FOO: 1.1:1,000000 de_DE.UTF-8
> > > 1
> > > - so far only macOS's system perl seems to be affected. A perl 5.29.9
> > > compiled myself on the same system does not exhibit the problem and
> > > neither does a Debian testing box.
> I can reproduce it on RHEL 7 with DBD::Pg git head.
That's a relief for sure. :)
> That strtod() call is a
> somewhat-recent addition (added in DBD::Pg 3.6.0, from 2017). What is the
> output of these two commands with each of those Perl installations?
> perl -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"'
> LANG=3Dde_DE.UTF-8 perl -MPOSIX -e 'use strict; use warnings; setlocale=
(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
Here's the output plus some unsolicited version info:
# /usr/bin/perl -V
Summary of my perl5 (revision 5 version 18 subversion 4) configuration:
=20
Platform:
osname=3Ddarwin, osvers=3D18.0, archname=3Ddarwin-thread-multi-2level
uname=3D'darwin osx316.apple.com 18.0 darwin kernel version 17.0.0: fri=
may 4 10:33:38 pdt 2018; root:xnu-4570.1.46.100.2~1development_x86_64 x86_=
64 '
config_args=3D'-ds -e -Dprefix=3D/usr -Dccflags=3D-g -pipe -Dldflags=
=3D -Dman3ext=3D3pm -Duseithreads -Duseshrplib -Dinc_version_list=3Dnone -D=
cc=3Dcc'
hint=3Drecommended, useposix=3Dtrue, d_sigaction=3Ddefine
useithreads=3Ddefine, usemultiplicity=3Ddefine
useperlio=3Ddefine, d_sfio=3Dundef, uselargefiles=3Ddefine, usesocks=3D=
undef
use64bitint=3Ddefine, use64bitall=3Ddefine, uselongdouble=3Dundef
usemymalloc=3Dn, bincompat5005=3Dundef
Compiler:
cc=3D'cc', ccflags =3D' -g -pipe -fno-common -DPERL_DARWIN -fno-strict-=
aliasing -fstack-protector',
optimize=3D'-Os',
cppflags=3D'-g -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fs=
tack-protector'
ccversion=3D'', gccversion=3D'4.2.1 Compatible Apple LLVM 10.0.1 (clang=
-1001.0.37.14)', gccosandvers=3D''
intsize=3D4, longsize=3D8, ptrsize=3D8, doublesize=3D8, byteorder=3D123=
45678
d_longlong=3Ddefine, longlongsize=3D8, d_longdbl=3Ddefine, longdblsize=
=3D16
ivtype=3D'long', ivsize=3D8, nvtype=3D'double', nvsize=3D8, Off_t=3D'of=
f_t', lseeksize=3D8
alignbytes=3D8, prototype=3Ddefine
Linker and Libraries:
ld=3D'cc', ldflags =3D' -fstack-protector'
libpth=3D/usr/lib /usr/local/lib
libs=3D=20
perllibs=3D
libc=3D, so=3Ddylib, useshrplib=3Dtrue, libperl=3Dlibperl.dylib
gnulibc_version=3D''
Dynamic Linking:
dlsrc=3Ddl_dlopen.xs, dlext=3Dbundle, d_dlsymun=3Dundef, ccdlflags=3D' '
cccdlflags=3D' ', lddlflags=3D' -bundle -undefined dynamic_lookup -fsta=
ck-protector'
Characteristics of this binary (from libperl):=20
Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
PERL_PRESERVE_IVUV PERL_SAWAMPERSAND USE_64_BIT_ALL
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
USE_REENTRANT_API
Locally applied patches:
/Library/Perl/Updates/<version> comes before system perl directories
installprivlib and installarchlib points to the Updates directory
Built under darwin
Compiled at Apr 1 2019 13:12:58
@INC:
/Library/Perl/5.18/darwin-thread-multi-2level
/Library/Perl/5.18
/Network/Library/Perl/5.18/darwin-thread-multi-2level
/Network/Library/Perl/5.18
/Library/Perl/Updates/5.18.4
/System/Library/Perl/5.18/darwin-thread-multi-2level
/System/Library/Perl/5.18
/System/Library/Perl/Extras/5.18/darwin-thread-multi-2level
/System/Library/Perl/Extras/5.18
.
# /usr/bin/perl -I/tmp/dbdpg/lib/perl5 -MDBD::Pg -e 'print $DBD::Pg::VERSIO=
N, "\n"'
3.7.4
# /usr/bin/perl -MPOSIX -e 'use strict; use warnings; setlocale(LC_NUMERIC,=
""); print join " ", strtod("1.1"), "\n"'
1.1 0=20
# LANG=3Dde_DE.UTF-8 /usr/bin/perl -MPOSIX -e 'use strict; use warnings; se=
tlocale(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1 2=20
# /tmp/myperl/bin/perl5.29.9 -V
Summary of my perl5 (revision 5 version 29 subversion 9) configuration:
=20
Platform:
osname=3Ddarwin
osvers=3D18.5.0
archname=3Ddarwin-thread-multi-2level
uname=3D'darwin mac.fritz.box 18.5.0 darwin kernel version 18.5.0: mon =
mar 11 20:40:32 pdt 2019; root:xnu-4903.251.3~3release_x86_64 x86_64 '
config_args=3D'-ds -e -Dprefix=3D/tmp/myperl -Dcc=3Dclang -Dccflags=3D =
-mmacosx-version-min=3D10.11 -isysroot /Applications/Xcode.app/Contents/Dev=
eloper/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -pipe -fno-=
common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector-strong -Adefine=
:cppflags=3D-no-cpp-precomp -mmacosx-version-min=3D10.11 -isysroot /Applica=
tions/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs=
/MacOSX10.11.sdk -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fsta=
ck-protector-strong -Adefine:ld=3Dclang -Aappend:ldflags=3D -Wl,-syslibroot=
,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Devel=
oper/SDKs/MacOSX10.11.sdk -mmacosx-version-min=3D10.11 -Aappend:lddlflags=
=3D -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/Ma=
cOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=3D10.11 -=
Dman3ext=3D3pm -Duseithreads -Dinc_version_list=3Dnone -Duserelocatableinc =
-Dusedevel'
hint=3Drecommended
useposix=3Dtrue
d_sigaction=3Ddefine
useithreads=3Ddefine
usemultiplicity=3Ddefine
use64bitint=3Ddefine
use64bitall=3Ddefine
uselongdouble=3Dundef
usemymalloc=3Dn
default_inc_excludes_dot=3Ddefine
bincompat5005=3Dundef
Compiler:
cc=3D'clang'
ccflags =3D'-mmacosx-version-min=3D10.11 -isysroot /Applications/Xcode.=
app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11=
=2Esdk -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protect=
or-strong -DPERL_USE_SAFE_PUTENV'
optimize=3D'-O3'
cppflags=3D'-no-cpp-precomp -mmacosx-version-min=3D10.11 -isysroot /App=
lications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/=
SDKs/MacOSX10.11.sdk -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -=
fstack-protector-strong'
ccversion=3D''
gccversion=3D'4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.3)'
gccosandvers=3D''
intsize=3D4
longsize=3D8
ptrsize=3D8
doublesize=3D8
byteorder=3D12345678
doublekind=3D3
d_longlong=3Ddefine
longlongsize=3D8
d_longdbl=3Ddefine
longdblsize=3D16
longdblkind=3D3
ivtype=3D'long'
ivsize=3D8
nvtype=3D'double'
nvsize=3D8
Off_t=3D'off_t'
lseeksize=3D8
alignbytes=3D8
prototype=3Ddefine
Linker and Libraries:
ld=3D'clang'
ldflags =3D' -mmacosx-version-min=3D10.14 -Wl,-syslibroot,/Applications=
/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacO=
SX10.11.sdk -mmacosx-version-min=3D10.11 -fstack-protector-strong'
libpth=3D/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDef=
ault.xctoolchain/usr/lib/clang/10.0.1/lib /Applications/Xcode.app/Contents/=
Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /Applications/Xcode.a=
pp/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.=
sdk/usr/lib /usr/lib
libs=3D-lpthread -ldbm -ldl -lm -lutil -lc
perllibs=3D-lpthread -ldl -lm -lutil -lc
libc=3D
so=3Ddylib
useshrplib=3Dfalse
libperl=3Dlibperl.a
gnulibc_version=3D''
Dynamic Linking:
dlsrc=3Ddl_dlopen.xs
dlext=3Dbundle
d_dlsymun=3Dundef
ccdlflags=3D' '
cccdlflags=3D' '
lddlflags=3D' -mmacosx-version-min=3D10.14 -bundle -undefined dynamic_l=
ookup -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/=
MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -mmacosx-version-min=3D10.11=
-fstack-protector-strong'
Characteristics of this binary (from libperl):=20
Compile-time options:
HAS_TIMES
MULTIPLICITY
PERLIO_LAYERS
PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP
PERL_OP_PARENT
PERL_PRESERVE_IVUV
PERL_USE_DEVEL
PERL_USE_SAFE_PUTENV
USE_64_BIT_ALL
USE_64_BIT_INT
USE_ITHREADS
USE_LARGE_FILES
USE_LOCALE
USE_LOCALE_COLLATE
USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO
USE_PERL_ATOF
USE_REENTRANT_API
USE_THREAD_SAFE_LOCALE
Built under darwin
Compiled at Apr 3 2019 22:40:04
@INC:
/tmp/myperl/lib/site_perl/5.29.9/darwin-thread-multi-2level
/tmp/myperl/lib/site_perl/5.29.9
/tmp/myperl/lib/5.29.9/darwin-thread-multi-2level
/tmp/myperl/lib/5.29.9
# /tmp/myperl/bin/perl5.29.9 -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"'
3.7.4
# /tmp/myperl/bin/perl5.29.9 -MPOSIX -e 'use strict; use warnings; setlocal=
e(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1.1 0=20
# LANG=3Dde_DE.UTF-8 /tmp/myperl/bin/perl5.29.9 -MPOSIX -e 'use strict; use=
warnings; setlocale(LC_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1 2=20
While this would seem to suggest that both use a locale-aware strtod in the
POSIX module, the reproducer still holds:
# /usr/bin/perl -I/tmp/dbdpg/lib/perl5 t.pl
1.1
# LANG=3Dde_DE.UTF-8 /usr/bin/perl -I/tmp/dbdpg/lib/perl5 t.pl
1
# /tmp/myperl/bin/perl5.29.9 t.pl
1.1
# LANG=3Dde_DE.UTF-8 /tmp/myperl/bin/perl5.29.9 t.pl
1.1
That would seem to suggest that DBD::Pg in my self-compiled perl uses anoth=
er
strtod than the POSIX module. The same seems to be the case on Debian testi=
ng:
# perl -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"'
3.9.1
# perl -MPOSIX -e 'use strict; use warnings; setlocale(LC_NUMERIC, ""); pri=
nt join " ", strtod("1.1"), "\n"'
1.1 0=20
# LANG=3Dde_DE.UTF-8 perl -MPOSIX -e 'use strict; use warnings; setlocale(L=
C_NUMERIC, ""); print join " ", strtod("1.1"), "\n"'
1 2=20
# perl t.pl
1.1
# LANG=3Dde_DE.UTF-8 perl t.pl
1.1
# perl -V
Summary of my perl5 (revision 5 version 28 subversion 1) configuration:
=20
Platform:
osname=3Dlinux
osvers=3D4.9.0
archname=3Dx86_64-linux-gnu-thread-multi
uname=3D'linux localhost 4.9.0 #1 smp debian 4.9.0 x86_64 gnulinux '
config_args=3D'-Dusethreads -Duselargefiles -Dcc=3Dx86_64-linux-gnu-gcc=
-Dcpp=3Dx86_64-linux-gnu-cpp -Dld=3Dx86_64-linux-gnu-gcc -Dccflags=3D-DDEB=
IAN -Wdate-time -D_FORTIFY_SOURCE=3D2 -g -O2 -fdebug-prefix-map=3D/build/pe=
rl-5WfRyb/perl-5.28.1=3D. -fstack-protector-strong -Wformat -Werror=3Dforma=
t-security -Dldflags=3D -Wl,-z,relro -Dlddlflags=3D-shared -Wl,-z,relro -Dc=
ccdlflags=3D-fPIC -Darchname=3Dx86_64-linux-gnu -Dprefix=3D/usr -Dprivlib=
=3D/usr/share/perl/5.28 -Darchlib=3D/usr/lib/x86_64-linux-gnu/perl/5.28 -Dv=
endorprefix=3D/usr -Dvendorlib=3D/usr/share/perl5 -Dvendorarch=3D/usr/lib/x=
86_64-linux-gnu/perl5/5.28 -Dsiteprefix=3D/usr/local -Dsitelib=3D/usr/local=
/share/perl/5.28.1 -Dsitearch=3D/usr/local/lib/x86_64-linux-gnu/perl/5.28.1=
-Dman1dir=3D/usr/share/man/man1 -Dman3dir=3D/usr/share/man/man3 -Dsiteman1=
dir=3D/usr/local/man/man1 -Dsiteman3dir=3D/usr/local/man/man3 -Duse64bitint=
-Dman1ext=3D1 -Dman3ext=3D3perl -Dpager=3D/usr/bin/sensible-pager -Uafs -U=
d_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly -D=
DEBUGGING=3D-g -Doptimize=3D-O2 -dEs -Duseshrplib -Dlibperl=3Dlibperl.so.5.=
28.1'
hint=3Drecommended
useposix=3Dtrue
d_sigaction=3Ddefine
useithreads=3Ddefine
usemultiplicity=3Ddefine
use64bitint=3Ddefine
use64bitall=3Ddefine
uselongdouble=3Dundef
usemymalloc=3Dn
default_inc_excludes_dot=3Ddefine
bincompat5005=3Dundef
Compiler:
cc=3D'x86_64-linux-gnu-gcc'
ccflags =3D'-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-ali=
asing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=3D=
64'
optimize=3D'-O2 -g'
cppflags=3D'-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-ali=
asing -pipe -I/usr/local/include'
ccversion=3D''
gccversion=3D'8.3.0'
gccosandvers=3D''
intsize=3D4
longsize=3D8
ptrsize=3D8
doublesize=3D8
byteorder=3D12345678
doublekind=3D3
d_longlong=3Ddefine
longlongsize=3D8
d_longdbl=3Ddefine
longdblsize=3D16
longdblkind=3D3
ivtype=3D'long'
ivsize=3D8
nvtype=3D'double'
nvsize=3D8
Off_t=3D'off_t'
lseeksize=3D8
alignbytes=3D8
prototype=3Ddefine
Linker and Libraries:
ld=3D'x86_64-linux-gnu-gcc'
ldflags =3D' -fstack-protector-strong -L/usr/local/lib'
libpth=3D/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed /=
usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /us=
r/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=3D-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=3D-ldl -lm -lpthread -lc -lcrypt
libc=3Dlibc-2.28.so
so=3Dso
useshrplib=3Dtrue
libperl=3Dlibperl.so.5.28
gnulibc_version=3D'2.28'
Dynamic Linking:
dlsrc=3Ddl_dlopen.xs
dlext=3Dso
d_dlsymun=3Dundef
ccdlflags=3D'-Wl,-E'
cccdlflags=3D'-fPIC'
lddlflags=3D'-shared -L/usr/local/lib -fstack-protector-strong'
Characteristics of this binary (from libperl):=20
Compile-time options:
HAS_TIMES
MULTIPLICITY
PERLIO_LAYERS
PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP
PERL_OP_PARENT
PERL_PRESERVE_IVUV
USE_64_BIT_ALL
USE_64_BIT_INT
USE_ITHREADS
USE_LARGE_FILES
USE_LOCALE
USE_LOCALE_COLLATE
USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO
USE_PERL_ATOF
USE_REENTRANT_API
Locally applied patches:
[...]
Built under linux
Compiled at Mar 31 2019 11:51:22
@INC:
/etc/perl
/usr/local/lib/x86_64-linux-gnu/perl/5.28.1
/usr/local/share/perl/5.28.1
/usr/lib/x86_64-linux-gnu/perl5/5.28
/usr/share/perl5
/usr/lib/x86_64-linux-gnu/perl/5.28
/usr/share/perl/5.28
/usr/local/lib/site_perl
/usr/lib/x86_64-linux-gnu/perl-base
> > > The text representation of values is whatever strings are
> > > produced and accepted by the input/output conversion functions
> > > for the particular data type.
> PostgreSQL always prints ".", not a locale-specific radix character. A
> locale-ignorant strtod() would suffice in DBD::Pg.
Awesome. Is there anyway I can help with this?
--=20
Thanks,
Michael