[Bug 296767] powerpc64le: panic in _vm_page_pqstate_commit_dequeue (page-queue TAILQ corruption) under heavy SMP load

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296767

            Bug ID: 296767
           Summary: powerpc64le: panic in _vm_page_pqstate_commit_dequeue
                    (page-queue TAILQ corruption) under heavy SMP load
           Product: Base System
           Version: 16.0-CURRENT
          Hardware: powerpc
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

Summary
-------
On powerpc64le, 16.0-CURRENT reliably panics in the VM page-queue code
(_vm_page_pqstate_commit_dequeue) under heavy parallel build load.  The page
queue's TAILQ linkage is found corrupted (a node's next->prev no longer points
back at the node), which INVARIANTS catches at the TAILQ_REMOVE in the
speculative-dequeue path.

Panic message
-------------
panic: /usr/src/sys/vm/vm_page.c:3716: _vm_page_pqstate_commit_dequeue:
       Bad link elm 0xc0000079936c7cc0 next->prev != elm

Environment
-----------
FreeBSD 16.0-CURRENT #1 main-n287158-c9991e01149b: Wed Jul  1 10:39:39 2026
  arch:      powerpc64le (__FreeBSD_version 1600019)
  kernel:    GENERIC64LE, options INVARIANTS + WITNESS + WITNESS_SKIPSPIN
  hardware:  IBM POWER9, hw.ncpu = 96, 512 GB RAM
  platform:  KVM guest (virtio: vtbd/vtnet) on a Talos II POWER9 host, UFS root

Reproduction (on demand, 5/5)
-----------------------------
Drive heavy, highly parallel VM page churn.  Every occurrence here was during a
poudriere build of lang/gcc14 with make parallelism enabled (ALLOW_MAKE_JOBS),
which saturates all 96 cores -- 1-minute load average ~98 -- during the gcc
bootstrap compile.  The box panics partway through that compile every time; it
never panics at idle or under light load.  Five consecutive attempts each
panicked at the same assertion (only the elm pointer differs).

Analysis
--------
vm_page.c:3716 is the TAILQ_REMOVE(&pq->pq_pl, m, plinks.q) inside
_vm_page_pqstate_commit_dequeue(), in the speculative-removal path guarded by
"(old->flags & PGA_ENQUEUED) != 0".  The comment immediately above it is
suggestive:

    /*
     * Once the queue index of the page changes there is nothing
     * synchronizing with further updates to the page's physical
     * queue state.  Therefore we must speculatively remove the page
     * from the queue now and be prepared to roll back if the queue
     * state update fails.  ...
     */

The corrupted back-link (next->prev != elm) indicates the pq_pl TAILQ was
modified concurrently with this removal.  That it reproduces only on
powerpc64le
under maximal SMP page churn, and not at low concurrency, is consistent with a
synchronization/memory-ordering race in the page-queue commit path that the
weaker POWER memory model exposes where amd64's TSO does not.  This is offered
as a lead, not a conclusion -- the exact call path is still needed.

Debugging status
----------------
A clean minidump was captured (see below) but is not yet symbolized: devel/gdb
15.1's kgdb loads the kernel and kernel.debug fine but rejects the dump with
"Failed to open vmcore: invalid corefile", so no backtrace is available.  This
looks like a gdb/ppc64le kernel-minidump limitation and may warrant its own PR.

  Dump header (info): Architecture powerpc64le, Dump Length 1040429056,
  Compression none, Dump Status good.

vmcore.4 (~1 GB) + the matching /boot/kernel/kernel and kernel.debug are
preserved and can be provided.  The reproducer is fast and deterministic, so
additional instrumentation (e.g. QUEUE_MACRO_DEBUG_TRASH, extra asserts, or a
targeted patch) can be turned around quickly.

-- 
You are receiving this mail because:
You are the assignee for the bug.