https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297767
Bug ID: 297767
Summary: Linuxulator amd64 vDSO collision with Android/Bionic
linker causes protection fault
Product: Base System
Version: 15.1-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
Created attachment 274003
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=274003&action=edit
strace excerpt
# FreeBSD Linuxulator: Android/Bionic x86-64 startup failure
## Environment
FreeBSD 15.1-RELEASE-p2 GENERIC amd64. Linuxulator: stock linux64.ko.
linux_base-rl9-9.7 installed.
## Binary under test
Any Android NDK x86-64 PIE executable, dynamically linked against Bionic libc
with interpreter /system/bin/linker64. The crash occurs in Bionic's `_start`
initialization (auxv walking, vDSO interaction) before `main()` is reached, so
the specific application code is irrelevant. The original investigation used a
more complex binary, but a trivial `int main(){return 0;}` compiled with the
NDK hits the same path.
## Primary bug: vDSO/stack layout collision
### Symptom
Bionic linker64 loads all shared libraries, completes signal handler setup and
RELRO finalization, and begins executing init_array constructors. During this
early startup phase (before `main()` is reached), a write to 0x7fffffffe950
faults because Linuxulator has mapped that region as r-x (the Linux vDSO).
### Faulting instruction
48 89 07 = mov [rdi], rax (WRITE)
Register state at fault (captured via ptrace tracer):
RIP = 0x801040535 (Bionic runtime, libc.so)
RDI = 0x7fffffffe950 (fault address, in vDSO region)
RAX = 0x7fffffffed00 (value: stack pointer being stored)
RSP = 0x7fffffffc860 (in rw- stack region)
RBP = 0x950
RSI = 0x80104faac (source data pointer)
RDX = 0x8 (count)
RFLAGS = 0x10206
### Memory layout at fault
From procstat vm of the crashing process:
0x7fffdfffd000 - 0x7ffffffdd000 --- guard page
0x7ffffffdd000 - 0x7fffffffd000 rw- stack (128KB)
0x7fffffffd000 - 0x7fffffffe000 r-x vDSO page 1
0x7fffffffe000 - 0x800000000000 r-x vDSO pages 2-3
Fault address 0x7fffffffe950 falls within the r-x vDSO mapping at offset 0x950.
### Strace evidence
Full strace of child process shows successful library loading through linker64
startup:
execve(/system/bin/main, ...) = 0
set_tid_address(...) = pid
arch_prctl(ARCH_SET_FS, ...) = 0
sigaltstack(...) = 0
rt_sigaction(SIGABRT/SIGBUS/SIGSEGV/...) = 0
[... shared libraries load via mmap, mprotect finalizes RELRO ...]
mprotect(0x103b000, 8192, PROT_READ) = 0 <-- last syscall before fault
--- SIGBUS {si_signo=SIGBUS, si_code=0xc, si_addr=0x7fffffffe950} ---
After SIGBUS, Bionic handler enters tight loop: catch signal, try debuggerd
socket, fail, resume, fault again. With machdep.prot_fault_translation=2,
signal becomes SIGSEGV code=2 (SEGV_ACCERR), single clean crash.
### Root cause
Linuxulator constructs Linux initial process image via copyout_strings()
(sys/compat/linux/linux_elf.c:327) which writes auxv data starting from
PROC_PS_STRINGS(p) downward. The Linux vDSO is separately mapped at
LINUX_VDSOPAGE_LA48 (sys/compat/linux/linux_sysvec.c:83):
LINUX_VDSOPAGE_SIZE = PAGE_SIZE * 2
LINUX_VDSOPAGE_LA48 = VM_MAXUSER_ADDRESS_LA48 - LINUX_VDSOPAGE_SIZE
This places vDSO at 0x7fffffffe000-0x800000000000, mapped as r-x by
linux_map_vdso() (sys/compat/linux/linux_vdso.c:152) and exec_new_vmspace()
(sys/kern/kern_exec.c:1389).
Bionic startup code walks auxv-derived pointers and writes to addresses that
fall in the vDSO region. On real Linux, ELF loader constructs auxv and initial
stack as a unit, and vDSO placement is coordinated with stack layout so auxv
write targets stay in writable memory.
### Why making vDSO writable is not a fix
Experimental patch adding VM_PROT_WRITE to linux_map_vdso() let binary pass the
crash point and proceed further into Bionic initialization, confirming the vDSO
write is the failure point. But vDSO pages are shared across all Linux
processes. Writing to them corrupts vDSO code/data and panics the host kernel.
Observed during testing: machine became unresponsive, required hard reboot.
### Fix direction (open question)
The Linuxulator vDSO placement collides with writable address space Bionic
expects. Possible approaches:
- Relocate Linux vDSO below stack region or into mmap area (as Linux does for
some architectures)
- Adjust copyout_strings() to reserve space above auxv data not claimed by vDSO
mapping
- Coordinate exec_new_vmspace() shared page with linux_map_vdso() so writable
stack covers auxv targets
## Secondary findings
### 1. Signal code translation
vm_fault_trap() (sys/vm/vm_fault.c:786) for KERN_PROTECTION_FAILURE:
if (SV_CURPROC_ABI() == SV_ABI_FREEBSD && ...)
SIGSEGV/SEGV_ACCERR (for native)
else
SIGBUS/UCODE_PAGEFLT (for Linuxulator, UCODE_PAGEFLT = T_PAGEFLT = 12)
Linuxulator gets SIGBUS si_code=0xc, which is not a valid Linux SIGBUS code.
Linux defines BUS_ADRALN=1, BUS_ADRERR=2, BUS_OBJERR=3. The si_code 0xc is the
raw FreeBSD trap code leaking through.
sigbus_sicode2lsicode() in sys/compat/linux/linux_signal.c:624 passes unknown
codes through without translation.
Effect: Bionic handler interprets invalid code incorrectly, enters infinite
retry loop. sysctl machdep.prot_fault_translation=2 forces SIGSEGV mode for all
processes, producing single clean crash.
### 2. rseq syscall returns ENOSYS
linux_rseq() in sys/compat/linux/linux_rseq.c is a stub returning ENOSYS.
Bionic calls rseq() during startup; ENOSYS triggers a fallback path that
busy-waits with FUTEX_WAKE_PRIVATE at ~76K calls/sec. An experimental
implementation accepting registration eliminated the spin but caused kernel
panics when combined with other patches. Not a blocker for the vDSO crash but
affects startup performance.
### 3. Missing RNDGETENTCNT ioctl
Binary opens /dev/urandom and calls ioctl(fd, RNDGETENTCNT, &count).
Linuxulator passes through to FreeBSD which returns EINVAL. Standalone patch
adding case 0x5200 to linux_ioctl_fallback() resolved this but caused hangs
when combined with vDSO patch.
### 4. O_NOFOLLOW / devfs symlink
Bionic opens /dev/urandom with O_NOFOLLOW. FreeBSD devfs creates urandom as
symlink to random. Open fails with EMLINK (errno 31). Hand-created mknod device
nodes on ZFS/tmpfs return 0 bytes on read (only devfs-created nodes work with
the random driver). Disabling O_NOFOLLOW translation entirely caused kernel
panics.
## Reproducer binary
Any Android NDK r23b+ x86-64 binary linked against Bionic reproduces the crash.
Minimal build:
1. Download Android NDK r23b: https://developer.android.com/ndk/downloads
2. Compile:
echo 'int main(){return 0;}' > t.c
$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/x86_64-linux-android22-clang
t.c -o main
3. Copy the Bionic runtime files from the NDK sysroot into a rootfs directory:
mkdir -p rootfs/system/bin rootfs/system/lib64
cp main rootfs/system/bin/
cp
$NDK/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libc.so
rootfs/system/lib64/
cp
$NDK/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libm.so
rootfs/system/lib64/
cp
$NDK/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libdl.so
rootfs/system/lib64/
# Copy linker64 from NDK (platforms/android-22/arch-x86_64/usr/lib/linker64
or equivalent)
The Linuxulator's chroot must contain the linker64 and the shared libraries the
binary links against, at the paths the linker64 expects (/system/bin/ and
/system/lib64/).
## Stock-kernel reproducer
After building the binary and preparing rootfs as described above:
# Load Linuxulator
kldload linux64
# Signal fix (avoids infinite Bionic retry loop)
sysctl machdep.prot_fault_translation=2
# Place linker64 where Linuxulator can find it
cp rootfs/system/bin/linker64 /compat/linux/system/bin/linker64
# Place shared libraries where linker64 will find them
mkdir -p /compat/linux/system/lib64
cp rootfs/system/lib64/*.so /compat/linux/system/lib64/
# Brandelf as Linux (if built outside NDK sysroot)
brandelf -t Linux rootfs/system/bin/main
# Run
rootfs/system/bin/main
Expected: SIGSEGV code=2 (SEGV_ACCERR) at 0x7fffffffe950. Machine stays stable.
Without prot_fault_translation=2: infinite SIGBUS/retry loop (Bionic catches
invalid signal code, retries, faults again).
## Summary
| Bug | Behavior | Impact |
|-----|----------|--------|
| vDSO/stack layout | SIGSEGV at 0x7fffffffe950 | Blocks all Bionic binaries |
| SIGBUS translation | SIGBUS code=0xc not SIGSEGV code=2 | Infinite retry loop
|
| rseq ENOSYS | Futex spin at 76K/sec | Performance only |
| RNDGETENTCNT | ioctl EINVAL | Post-crash path |
| O_NOFOLLOW/devfs | EMLINK on urandom | Post-crash path |
First two bugs together prevent any Android/Bionic x86-64 binary from running
under Linuxulator.
------
# Strace excerpt: Bionic x86-64 binary startup under FreeBSD Linuxulator
# Child process (pid 4397) after execve(/system/bin/main)
# machdep.prot_fault_translation=2 (SIGSEGV mode)
# === linker64 startup (lines 54-68) ===
[pid 4397] execve("/system/bin/main", ["./wrapper", "-H", "0.0.0.0"],
0x7fffffffcc48 /* 16 vars */) = 0
[pid 4397] mprotect(0x801053000, 4096, PROT_READ) = 0
[pid 4397] set_tid_address(0x8010556b0) = 4397
[pid 4397] arch_prctl(ARCH_SET_FS, 0x801055200) = 0
[pid 4397] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x801057000
[pid 4397] sigaltstack({ss_sp=0x801057000, ss_flags=0, ss_size=8192}, NULL) =
0
[pid 4397] rt_sigaction(SIGABRT, {sa_handler=0x80103e8b0, ...}, NULL, 8) = 0
[pid 4397] rt_sigaction(SIGBUS, {sa_handler=0x80103e8b0, ...}, NULL, 8) = 0
[pid 4397] rt_sigaction(SIGSEGV, {sa_handler=0x80103e8b0, ...}, NULL, 8) = 0
[pid 4397] prctl(PR_SET_VMA, ..., "linker_alloc") = -1 EINVAL (Invalid
argument)
# === shared library loading (abbreviated; ~20 .so files) ===
[pid 4397] openat(AT_FDCWD, "/system/lib64/libc++_shared.so",
O_RDONLY|O_CLOEXEC) = 3
[pid 4397] mmap(NULL, 952680, PROT_READ|PROT_EXEC, ...) = 0x80105a000
[pid 4397] openat(AT_FDCWD, "/system/lib64/libc.so", O_RDONLY|O_CLOEXEC) = 4
[pid 4397] mmap(NULL, 855256, PROT_READ|PROT_EXEC, ...) = 0x801152000
[pid 4397] openat(AT_FDCWD, "/system/lib64/libandroidappmusic.so",
O_RDONLY|O_CLOEXEC) = 3
[pid 4397] mmap(NULL, 6015640, PROT_READ|PROT_EXEC, ...) = 0x80123d000
[... 17 more .so files loaded similarly ...]
# === RELRO finalization (last syscall before crash) ===
[pid 4397] mprotect(0x103b000, 8192, PROT_READ) = 0
[pid 4397] munmap(0x801059000, 4096) = 0
[pid 4397] close(3) = 0
# === SIGSEGV fault (with prot_fault_translation=2) ===
# Binary attempts: mov [0x7fffffffe950], rax
# Address 0x7fffffffe950 is in the r-x vDSO region
# With stock kernel (no sysctl): delivered as SIGBUS si_code=0xc (invalid)
# -> Bionic handler loops: catch signal, try debuggerd, fail, resume, fault
again
# With prot_fault_translation=2: delivered as SIGSEGV si_code=2 (SEGV_ACCERR)
# -> Bionic handler catches once, logs, resets SIGSEGV to SIG_DFL, resumes
# === post-crash: Bionic continues, reaches urandom open ===
[pid 4397] openat(AT_FDCWD, "/dev/__properties__",
O_RDONLY|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT
[pid 4397] futex(0x80122c6e0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid 4397] mprotect(0x8084b2000, 4096, PROT_READ|PROT_WRITE) = 0
[... 20 mprotect calls toggling RELRO segments ...]
[pid 4397] openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_NOFOLLOW|O_CLOEXEC) =
-1 ELOOP (Too many levels of symbolic links)
[pid 4397] tgkill(4397, 4397, SIGKILL) = 0
[pid 4397] --- SIGKILL {si_signo=SIGKILL, si_code=SI_USER, si_pid=0, si_uid=0}
---
[pid 4397] +++ killed by SIGKILL +++
--
You are receiving this mail because:
You are the assignee for the bug.
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.