bug#35245: libltdl.so uses 32-bit search path in 64-bit mode in Solaris 11.4

Hi all,

I noticed that 64-bit version of libtool's libltdl.so shipped in Solaris 11.4
is compiled with 32-bit search path. This mismatch leads causes a failure to
dynamically load a module using lt_dlopen() interface.

Here's an example of a 64-bit executable trying to dynamically load
"libcrypt.so":

$ truss -t open,mmapobj ./ltdldemo
openat(AT_FDCWD, "/var/ld/64/ld.config", O_RDONLY) Err#2 ENOENT
openat(AT_FDCWD, "/usr/lib/64/libltdl.so.7", O_RDONLY) = 3
openat(AT_FDCWD, "/lib/64/libc.so.1", O_RDONLY) = 3
calling lt_dlinit
calling lt_dlopen
openat(AT_FDCWD, "/usr/lib/libcrypt.so", O_RDONLY) = 3

^ found "libcrypt.so" in 32-bit path

mmapobj(3, MMOBJ_INTERPRET, 0x7FFFBF6ED210, 0x7FFFBFFFEA0C, 0x00000000) Err#48 ENOTSUP

^ failed to map it

libcrypt.so: file not found

I assume the failure is obvious. A 64-bit program doesn't like 32-bit
library (/usr/lib/libcrypt.so) and so mmapobj(2) syscall returns ENOTSUP
per its man page:

       ENOTSUP       The current user data model does not match the fd  to  be
                     interpreted.  For example, a 32-bit process that tried to
                     use mmapobj() to interpret a 64-bit object  would  return

The fix is to use 64-bit default search path in 64-bit mode (/lib/64
/usr/lib/64). There are two workarounds:

1. LTDL_LIBRARY_PATH=/lib/64:/usr/lib/64
2. re-compile 64-bit libtool:
     ./configure lt_cv_sys_dlsearch_path="/lib/64 /usr/lib/64"

Perhaps the best solution would be to auto-detect whether to use 32-bit or
64-bit library search paths [1][2][3] on Solaris OS in a configure script.

Please see a suggested fix in an attached patch.

Thanks,

Vita

[1] Search paths per ld(1) man page:
       LIBPATH        For  32-bit  libraries, the default search path is /lib,
                      followed by /usr/lib. For 64-bit libraries, the  default
                      search path is /lib/64, followed by /usr/lib/64.

[2] Search paths per ld.so.1(1) man page:
       The runtime linker uses a  prescribed  search  path  for  locating  the
       dynamic  dependencies  of  an  object. The default search paths are the
       runpath recorded in the object, followed by a series of  defaults.  For
       32-bit  objects, the defaults are /lib followed by /usr/lib. For 64-bit
       objects, the  defaults  are  /lib/64  followed  by  /usr/lib/64.

[3] Search paths per crle(1) man page:
           The  default  search paths for 32-bit ELF objects are /lib followed
           by /usr/lib. For 64-bit ELF objects, the default search  paths  are
           /lib/64 followed by /usr/lib/64.

_______________________________________________
Bug-libtool mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/bug-libtool

Prevent libltdl.so from using 32-bit search path in 64-bit mode in Solaris OS.

--- libtool-2.4.6.orig/m4/libtool.m4	Tue Jan 20 17:15:19 2015
+++ libtool-2.4.6/m4/libtool.m4	Thu Apr 11 19:20:11 2019
@@ -1456,6 +1456,7 @@
   if AC_TRY_EVAL(ac_compile); then
     case `/usr/bin/file conftest.o` in
     *64-bit*)
+      SOLARIS_ISA_BITS=64
       case $lt_cv_prog_gnu_ld in
       yes*)
         case $host in
@@ -1478,6 +1479,9 @@
 	;;
       esac
       ;;
+    *32-bit*)
+      SOLARIS_ISA_BITS=32
+      ;;
     esac
   fi
   rm -rf conftest*
@@ -2995,6 +2999,12 @@
   hardcode_into_libs=yes
   # ldd complains unless libraries are executable
   postinstall_cmds='chmod +x $lib'
+  if test 32 = "$SOLARIS_ISA_BITS"; then
+    sys_lib_search_path_spec="/lib /usr/lib"
+  else
+    sys_lib_search_path_spec="/lib/64 /usr/lib/64"
+  fi
+  sys_lib_dlsearch_path_spec=$sys_lib_search_path_spec
   ;;
 
 sunos4*)