Re: rump_server, "panic: uvm_km_alloc failed" on module load

I've managed to make rump_server -m working on amd64. In general
rump_server needs same statically linked libraries as t_modautoload.

Complete Makefile which produces working rump_server:

--------------------
#	$NetBSD: Makefile,v 1.18 2024/04/20 13:24:49 rillig Exp $
#

NOFULLRELRO=	yes

.PATH: ${.CURDIR}/../rump_allserver

PROG=		rump_server
SRCS=		rump_allserver.c
NOMAN=		installed by ../rump_allserver
NOLINT=		# LDADD contains -Wl,...

# Needs executable and writable mmap() when loading modules
PAXCTL_FLAGS=	+ma


# We need to be able to find our own statically linked symbols when linking
# in modules
LDFLAGS+=	-Wl,-E


# Note: we link the rump kernel into the application to make this work
# on amd64.

# If librump is not linked in then calls to in kernel functions in rump
# space don't get resolved properly

# All of these have to be static, otherwise linker gets really angry
# starting with can not find libgcc_s.a

LDADD+= \
     -Wl,--whole-archive \
	-Wl,-Bstatic \
	-lrump_g \
	-lrumpvfs_g \
	-lrumpvfs_nofifofs_g \
	-lrumpkern_sysproxy_g \
     -Wl,-Bdynamic -Wl,--no-whole-archive

LDADD+=	 -lpthread  -lrumpuser

.if ${RUMP_SANITIZE:Uno} != "no"
LDADD+=	-fsanitize=${RUMP_SANITIZE}
.endif

.include <bsd.prog.mk>
-----------------------


This breaks loadig of libraries which are already statically linked :
./rump_server -s -v -lrumpvfs 
-m/stand/amd64/10.99.15/modules/tmpfs/tmpfs.kmod   unix://sock

Or example from man page:
$ rump_server -lrumpvfs -m /modules/tmpfs.kmod unix://sock

This works as expected:
./rump_server -s -v -lrumpnet 
-m/stand/amd64/10.99.15/modules/tmpfs/tmpfs.kmod   unix://sock







On 8/1/25 14:08, Christoph Badura wrote:
> 
> I haven't done this before, so no idea if this helps.
> 
> gdb has an add-symbol-file command.  And
> $OBJDIR/sys/modules/hello/hello/kmod.debug should have symbols.  Note that
> that file probably is not installed to /stand.
> 
> add-symbol-file needs the address where the .text section was loaded.  I
> don't know how to get that.
> 
> However, hello.c has only calls to printf().  Perhaps the references to that
> function are not correctly resolved?  Or maybe they are not resolved to the
> correct function.  This should be calling the kernel printf which might
> have been renamed to rumpns_printf?  But maybe it doesn't make a
> difference.
> 

Thanks to tip from Matthew I was able to load debug symbols for modules
in gdb:

Breakpoint after module is loaded but before call to module
init (kern_module.c:1420)

set $ko = mod->mod_kobj
set $n = mod->mod_info.mi_name
eval "add-symbol-file %s/%s/%s.kmod -s .text 0x%lx -s .data 0x%lx -s 
.rodata 0x%lx\n", module_base, $n, $n, $ko->ko_text_address, 
$ko->ko_data_address, $ko->ko_rodata_address


With that it is possible to set breakpoints inside module. Inspecting
address functions for t_modautoload and rump_server without static
libraries I get this addresses for functions:


t_modautoload
(gdb) print vfs_attach
$3 = {int (struct vfsops *)} 0x9aa57 <vfs_attach>

(gdb) print printf
$4 = {void (const char *, ...)} 0x107f8e <printf>


rump_server
(gdb) print vfs_attach
$3 = {int (struct vfsops *)} 0x7f7ff7d1885a <vfs_attach>

(gdb) print printf
$1 = {void (const char *, ...)} 0x7f7ff7deca42 <printf>


Module is loaded at 0x7f7ff777fd00.

(gdb) info sharedlibrary
 From                To                  Syms Read   Shared Object Library
0x00007f7ff7eed000  0x00007f7ff7ef84f7  Yes         /usr/libexec/ld.elf_so
0x00007f7ff7edf150  0x00007f7ff7edf930  Yes 
/usr/lib/librumpkern_sysproxy.so.0
0x00007f7ff7d90320  0x00007f7ff7e72d00  Yes         /usr/lib/librump.so.0
0x00007f7ff7d4c040  0x00007f7ff7d4c0fc  Yes 
/usr/lib/librumpvfs_nofifofs.so.0
0x00007f7ff7cc6d00  0x00007f7ff7d28edb  Yes         /usr/lib/librumpvfs.so.0
0x00007f7ff7c8b830  0x00007f7ff7c91633  Yes 
/usr/lib/librumpuser.so.0
0x00007f7ff7c77c80  0x00007f7ff7c7eb6b  Yes         /usr/lib/libpthread.so.1
0x00007f7ff7872550  0x00007f7ff79eab5d  Yes         /usr/lib/libc.so.12

And address of printf is correct:

(gdb) disassemble 0x7f7ff7deca42
Dump of assembler code for function printf:
    0x00007f7ff7deca42 <+0>:     push   %rbp
    0x00007f7ff7deca43 <+1>:     mov    %rsp,%rbp
    0x00007f7ff7deca46 <+4>:     sub    $0x60,%rsp
    0x00007f7ff7deca4a <+8>:     mov    %rdi,-0x58(%rbp)
    0x00007f7ff7deca4e <+12>:    mov    %rsi,-0x28(%rbp)
    0x00007f7ff7deca52 <+16>:    mov    %rdx,-0x20(%rbp)
    0x00007f7ff7deca56 <+20>:    mov    %rcx,-0x18(%rbp)
    0x00007f7ff7deca5a <+24>:    mov    %r8,-0x10(%rbp)
    0x00007f7ff7deca5e <+28>:    mov    %r9,-0x8(%rbp)
    0x00007f7ff7deca62 <+32>:    movl   $0x8,-0x48(%rbp)
    0x00007f7ff7deca69 <+39>:    lea    0x10(%rbp),%rax
    0x00007f7ff7deca6d <+43>:    mov    %rax,-0x40(%rbp)
    0x00007f7ff7deca71 <+47>:    lea    -0x30(%rbp),%rax
    0x00007f7ff7deca75 <+51>:    mov    %rax,-0x38(%rbp)
    0x00007f7ff7deca79 <+55>:    lea    -0x48(%rbp),%rdx
    0x00007f7ff7deca7d <+59>:    mov    -0x58(%rbp),%rax
    0x00007f7ff7deca81 <+63>:    mov    %rax,%rsi
    0x00007f7ff7deca84 <+66>:    mov    $0x5,%edi
    0x00007f7ff7deca89 <+71>:    call   0x7f7ff7d8d0b0 
<rumpns_vprintf_flags@plt>
    0x00007f7ff7deca8e <+76>:    nop
    0x00007f7ff7deca8f <+77>:    mov    %rbp,%rsp
    0x00007f7ff7deca92 <+80>:    pop    %rbp
    0x00007f7ff7deca93 <+81>:    ret
End of assembler dump.


But address of rumpns_vprintf_flags doesn't look right:

(gdb) disassemble 0x7f7ff7d8d0b0
Dump of assembler code for function rumpns_vprintf_flags@plt:
    0x00007f7ff7d8d0b0 <+0>:     jmp    *0x12b792(%rip)        # 
0x7f7ff7eb8848 <[email protected]>
    0x00007f7ff7d8d0b6 <+6>:     push   $0x109
    0x00007f7ff7d8d0bb <+11>:    jmp    0x7f7ff7d8c010
End of assembler dump.


In fact gdb thinks there is quite a few 16 byte functions there:

(gdb) x/64x 0x7f7ff7d8d0b0
0x7f7ff7d8d0b0 <rumpns_vprintf_flags@plt>:      0xff    0x25    0x92 
0xb7    0x12    0x00    0x68    0x09
0x7f7ff7d8d0b8 <rumpns_vprintf_flags@plt+8>:    0x01    0x00    0x00 
0xe9    0x50    0xef    0xff    0xff
0x7f7ff7d8d0c0 <rumpns_module_init@plt>:        0xff    0x25    0x8a 
0xb7    0x12    0x00    0x68    0x0a
0x7f7ff7d8d0c8 <rumpns_module_init@plt+8>:      0x01    0x00    0x00 
0xe9    0x40    0xef    0xff    0xff
0x7f7ff7d8d0d0 <rumpns_config_match@plt>:       0xff    0x25    0x82 
0xb7    0x12    0x00    0x68    0x0b
0x7f7ff7d8d0d8 <rumpns_config_match@plt+8>:     0x01    0x00    0x00 
0xe9    0x30    0xef    0xff    0xff
0x7f7ff7d8d0e0 <rump_thread_init@plt>:  0xff    0x25    0x7a    0xb7 
0x12    0x00    0x68    0x0c
0x7f7ff7d8d0e8 <rump_thread_init@plt+8>:        0x01    0x00    0x00 
0xe9    0x20    0xef    0xff    0xff



> 
> This looks better.  I was wondering why "/module.mod"?  And discovered
> this:
> https://nxr.netbsd.org/xref/src/usr.bin/rump_allserver/rump_allserver.c#411
> 
> I guess that should be "modarray[i]", otherwise with multiple "-m" it loads
> the first module multiple times, doesn't it?

Right, it should be "modarray[i]", tries to load first module multiple
times as it is now. Tested with 2 modules, loads ok.

Both modarray[0] and "define ETFSKEY /module.mod" look to me like this
was work in progress.



Seeing not that static linking is needed for rump module loading to
work on amd64 I tried testing rump_server -m on arm64, and it has other
problem:

$ rump_server -s -v -lrumpvfs -lrumpvfs_nofifofs -lrump -m 
/stand/evbarm/10.99.14/modules/kernfs/kernfs.kmod   unix://sock
[   1.0000050] root file system type: rumpfs
[   1.0000050] kern.module.path=/stand/evbarm/10.99.14/modules
[   1.0200050] warning: kernel ABI not supported on this arch
[   1.0200050] panic: kobj_reloc: not supported on this architecture
[   1.0200050] rump kernel halting...


   Nemanja