Re: compile and run 32bit programs on aarch64eb

On Tue, Nov 25, 2025 at 22:17:54 +0100, Christian Groessler wrote:

> I couldn't make much sense out of
> 
>      Since NetBSD 5.0 the base system has directly included support for
> 32-bit
>      compatibility by installing 32-bit libraries and dynamic linker into
>      /usr.  This includes compiler support for compiling 32-bit applications
>      on platforms where this is supported.
> 
> 
> Where under /usr would I put the 32bit dynamic libraries?
> 
> Also this sentence
> 
>      Before NetBSD 5.0 all of these files needed to be placed under
>      /emul/netbsd32.
> 
> irritated me.
> 
> After NetBSD 5.0 (means: _now_) I've needed to place them under
> /emul/netbsd32 (specifically I've copied 32bit versions of /lib /libexec
> /usr/lib /usr/libexec there). If they can live elsewhere under /usr it's not
> obvious to me where, and I didn't want to drop them into the host's 64bit
> /lib /libexec /usr/lib /usr/libexec directories, fearing that I'd mess up my
> system.

Yes, that man page could use some editorial help.  I think currently
if conflates two different use cases.

The first use case is /emul - where you can grab e.g. sparc sets (or a
ranodm sparc binary), and unpack them under /emul/sparc on a sparc64
machine.  Such binaries obviously refer to /lib/libc.so etc, and on a
sparc64 machines these references would not work as is (/lib/libc.so
is a sparc64 libc).  So when these sparc binaries are run, they are
handled by the kernel exec module that knows to first resolve path
names under /emul/sparc (think plan9 namespace that has /emul/sparc
bound at root before the host's root).

The second use case is compiling code on sparc64 with -m32.  We do
that as part of the _64-bit_ build and ship it as the base32 set.

$ uname -a
NetBSD omena 11.0_BETA NetBSD 11.0_BETA (GENERIC) #0: Mon Oct  6 16:11:59 UTC 2025  [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC amd64
$ cat hello.c
#include <stdio.h>
int main() { printf("hello, world\n"); return 0; }
$ cc -m32 hello.c -o hello
$ file hello
hello: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 11.0, with debug_info, not stripped
$ readelf -d hello | grep NEED
 0x00000001 (NEEDED)                     Shared library: [libc.so.12]
$ ldd hello
hello:
	-lc.12 => /usr/lib/i386/libc.so.12
$ ./hello
hello, world

Here the interpreter is set to /usr/libexec/ld.elf_so (which is 64-bit
host dynamic linker), but sys/compat/common/compat_util.c in
compat_elf_check_interp sepcial cases this with:

  /*
   * If the path is exactly "/usr/libexec/ld.elf_so", first
   * try to see if "/usr/libexec/ld.elf_so-<abi>" exists
   * and if so, use that instead.
   */

Again, in plan9 terms think that /usr/libexec/ld.elf_so-$abi is bound
as /usr/libexec/ld.elf_so

The library search is special cased by libexec/ld.elf_so/rtld.c
where _rtld_init does:

      _rtld_add_paths(execname, &_rtld_default_paths,
          RTLD_DEFAULT_LIBRARY_PATH);

#ifdef RTLD_ARCH_SUBDIR
       _rtld_add_paths(execname, &_rtld_default_paths,
           RTLD_DEFAULT_LIBRARY_PATH "/" RTLD_ARCH_SUBDIR);
#endif


-uwe