[Bug backtrace/34476] New: MIPS o32: backtrace broken through syscall stubs that adjust SP after the prologue

https://sourceware.org/bugzilla/show_bug.cgi?id=3D34476

            Bug ID: 34476
           Summary: MIPS o32: backtrace broken through syscall stubs that
                    adjust SP after the prologue
           Product: gdb
           Version: HEAD
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: backtrace
          Assignee: unassigned at sourceware dot org
          Reporter: orgads at gmail dot com
  Target Milestone: ---

Created attachment 16900
  --> https://sourceware.org/bugzilla/attachment.cgi?id=3D16900&action=3Ded=
it
Reproducer: call chain main -> f1 -> f2 -> stub

On MIPS o32, syscalls taking five or more arguments pass the extra arguments
on the stack. Libc syscall stubs therefore temporarily lower SP around the
actual `syscall` instruction, *after* the function prologue. uClibc's
`__syscall_ipc` (compiled C) is representative:

```
addiu   sp,sp,-8        # prologue
sw      s0,4(sp)
lw      v0,24(sp)       # two "non-prologue" instructions
lw      s0,28(sp)
addiu   sp,sp,-32       # temp area for the syscall's stack arguments
...
syscall                 # <- a blocked thread's pc is here
addiu   sp,sp,32
```

uClibc's hand-written `syscall.S` and musl's stack-argument syscall wrappers
have the same shape. None of them carry CFI, so gdb unwinds them with the
heuristic prologue analyzer, `mips32_scan_prologue` (gdb/mips-tdep.c). That
function permits at most one non-prologue instruction:

```c
      /* A jump or branch, or enough non-prologue insns seen?  If so,
         then we must have reached the end of the prologue by now.  */
      if (prev_delay_slot || non_prologue_insns > 1)
        break;
```

so for a thread whose pc is at/after the `syscall` instruction, the scan
stops at the two `lw`s and never sees the `addiu sp,sp,-32`. The computed
frame base is 32 bytes too low, the caller's saved-ra slot is read from the
wrong address (it lands in the outgoing-argument area), and the backtrace
degenerates into a garbage frame right after the libc stub.

This makes core dumps of blocked multi-threaded programs on MIPS o32
routers/embedded systems (uClibc and musl userlands, e.g. OpenWrt) largely
undebuggable: every thread blocked in `msgrcv`, `ppoll`, `futex` via
`syscall()`, etc. loses its call chain after frame #1.

## Reproducer (stock Debian packages)

`apt install gcc-mips-linux-gnu libc6-dev-mips-cross qemu-user gdb-multiarc=
h`

stub.S, main.c: see attachments; `stub` reproduces the uClibc stub shape.

```
$ mips-linux-gnu-gcc -O2 -g -static -o repro main.c stub.S
$ qemu-mips -g 1234 ./repro &
$ gdb-multiarch -batch -ex 'target remote :1234' \
    -ex 'break stub.S:17' -ex 'continue' -ex 'bt' -ex 'kill' ./repro
```

Actual:

```
#0  stub () at stub.S:17
#1  0x00400724 in f2 () at main.c:2
#2  0x0049a0ec in __init_array_start ()
Backtrace stopped: frame did not save the PC
```

(#1 is found only because `stub` is a leaf and $ra is live; #2 is garbage
read 32 bytes below the real saved-ra slot.)

Expected:

```
#0  stub () at stub.S:17
#1  0x00400724 in f2 () at main.c:2
#2  0x00400758 in f1 () at main.c:3
#3  0x00400554 in main () at main.c:4
```

Also observed on real-world core dumps from a MIPS32r2 (Lantiq GRX) system
running uClibc 0.9.33.2: in a core of a large multi-threaded process, every
thread blocked in `msgrcv` showed `#3 0x00000004 in ?? ()` (the `msgsz`
argument misread as a return address) instead of the real caller chain.

## Suggested fix

When scanning for frame computation (not when skipping the prologue), after
the prologue-end heuristic fires, continue scanning up to the current pc,
accumulating further SP adjustments. Apply the extra adjustment to the frame
base and to the recorded register save slots (same restart mechanism the
scanner already uses for alloca). Patch attached
(0001-gdb-mips-unwind-past-post-prologue-SP-adjustments-in.patch, includes a
gdb.arch testcase); with it, the reproducer above unwinds fully:

```
#0  stub () at stub.S:17
#1  0x00400724 in f2 () at main.c:2
#2  0x00400758 in f1 () at main.c:3
#3  0x00400554 in main () at main.c:4
```

and on the real-world core, garbage frames across all 58 threads went from
32 to 0 (including a thread blocked in ppoll, whose syscall window is 216
bytes into the function - past the prologue scanner's 200-byte cap).

## Related bugs (searched, none is this issue)

- PR 20406 - microMIPS heuristic prologue scanner missing the alloca
  handling that mips32_scan_prologue has. Same area; also a reminder that
  the microMIPS and MIPS16 scanners would need the equivalent of this fix
  (the attached patch covers mips32_scan_prologue only).
- PR 11348 - broken backtrace from a MIPS signal handler (gdb 6.6, 2010).
  Superficially similar symptom, different mechanism: that one failed at
  the signal-trampoline crossing (sigframe unwinder); verified working on
  current master. Note that a signal interrupting a thread blocked in a
  stack-argument syscall reproduces *this* bug right after the
  "<signal handler called>" frame.
- PR 8823 - i386 prologue analyzer losing track of a scheduled prologue
  (fixed in 2004); the same class of heuristic-scanner limitation on
  another architecture.

## References: the SP dance in libc sources

The temporary SP adjustment around `syscall` is spelled out in the libc
sources for every o32 syscall with stack arguments (5+ args):

- uClibc 0.9.33.2, `internal_syscall5/6/7`
  (`subu $29, 32 ... sw args 16/20/24($29) ... syscall ... addiu $29, 32`):
=20
https://github.com/kraj/uClibc/blob/v0.9.33.2/libc/sysdeps/linux/mips/bits/=
syscalls.h#L186-L253
- uClibc 0.9.33.2, hand-written generic `syscall ()`:
=20
https://github.com/kraj/uClibc/blob/v0.9.33.2/libc/sysdeps/linux/mips/sysca=
ll.S#L53-L62
- uClibc-ng (current master), same macros:
=20
https://github.com/wbx-github/uclibc-ng/blob/master/libc/sysdeps/linux/mips=
/bits/syscalls.h#L196-L231
- musl (current master), `__syscall5/6/7`
  (`subu $sp,$sp,32 ... addu $sp,$sp,32`):
  https://git.musl-libc.org/cgit/musl/tree/arch/mips/syscall_arch.h#n96

Modern glibc is immune by design: since the switch to out-of-line
`__mips_syscall5/6/7` wrappers (sysdeps/unix/sysv/linux/mips/mips32/), the
stack arguments live in the compiled caller's frame and SP never moves in
the wrapper. Older glibc used the same inline-asm `subu $29, 32` pattern.

## Note: wrong CFI variant of the same problem

If the libc is built with -g, GCC on MIPS emits .debug_frame, and gdb then
prefers DWARF CFI over the prologue scanner - but that CFI is *also* wrong
for these stubs, because the SP adjustment happens inside an inline-asm
block GCC knows nothing about (hand-written CFI, as glibc does in its asm
stubs, is the only fix there). That variant is a libc issue, not a gdb one;
this report and patch address the common deployed case of stripped/CFI-less
libraries.

--=20
You are receiving this mail because:
You are on the CC list for the bug.=