Re: [PATCH v3 9/9] x86: re-enable EXECMEM_ROX support

On Wed, Aug 12, 2026 at 06:33:15AM +0000, Nikunj A Dadhania wrote:
> Hi,
> 
> A kernel BUG in __text_poke() is observed inside a Ubuntu 26.04 guest 
> (v7.0.0.28 and also with upstream v7.2-4c5). Host is running on v7.2-rc5.
> 
> SETUP
> -----
> 
> 512 QEMU Q35 guests (2 vCPUs, 1G RAM each) boot concurrently across 200
> iterations. The crash is seen during module loading in early boot.
> 2-7 guests out of ~100,000 boot attempts crash with the kernel BUG.
> 
> Reproduced on:
>   - AMD EPYC, QEMU Q35 guests
>   - Intel Icelake, QEMU Q35 guests
> 
> Apparently it was seen once in the non-guest Ubuntu 26.04 environment as well
> https://discourse.ubuntu.com/t/failed-boot-one-time-on-ubuntu-unity-26-04-educational-mission/85300
> 
> SYMPTOMS
> --------
> 
> Either of the below two crash signatures is observed:
> 
> Signature 1 — BUG_ON(!pages[0]) in __text_poke():
> 
>   kernel BUG at arch/x86/kernel/alternative.c:2564!
>   Oops: invalid opcode: 0000 [#1] SMP NOPTI
>   CPU: 0 UID: 0 PID: 741 Comm: (udev-worker) Not tainted 7.0.0-28-generic #28-Ubuntu PREEMPT(lazy)
>   Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)
>   RIP: 0010:__text_poke+0x2a4/0x400
>   Call Trace:
>    <TASK>
>    smp_text_poke_batch_finish+0x25f/0x5b0
>    smp_text_poke_single+0x13/0x30
>    __static_call_transform+0x10e/0x210
>    arch_static_call_transform+0x5c/0xc0
>    __static_call_init+0x154/0x300
>    static_call_module_notify+0x144/0x170
>    notifier_call_chain+0x62/0xf0
>    blocking_notifier_call_chain_robust+0x62/0xc0
>    load_module+0x5ef/0x960
>   Modules linked in: ppdev(+) kvm(+) irqbypass ...
> 
> Signature 2 — BUG_ON(memcmp()) in __text_poke():
> 
>   kernel BUG at arch/x86/kernel/alternative.c:2632!
>   Oops: invalid opcode: 0000 [#1] SMP NOPTI
>   CPU: 0 UID: 0 PID: 742 Comm: (udev-worker) Not tainted 7.0.0-28-generic #28-Ubuntu PREEMPT(lazy)
>   Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)
>   RIP: 0010:__text_poke+0x380/0x400
>   Call Trace:
>    <TASK>
>    smp_text_poke_batch_finish+0x323/0x5b0
>    smp_text_poke_single+0x13/0x30
>    __static_call_transform+0x10e/0x210
>    arch_static_call_transform+0x5c/0xc0
>    __static_call_init+0x154/0x300
>    static_call_module_notify+0x144/0x170
>    notifier_call_chain+0x62/0xf0
>    blocking_notifier_call_chain_robust+0x62/0xc0
>    load_module+0x5ef/0x960
>   Modules linked in: kvm(+) irqbypass ...
> 
> INSTRUMENTATION
> ---------------
> 
> The following instrumentation was applied to capture the page table state at
> the moment of failure:
> 
>   In mm/vmalloc.c::vmalloc_to_page()
> 
>     ptep = pte_offset_kernel(pmd, addr);
>     pte = ptep_get(ptep);
>     if (pte_present(pte))
>         page = pte_page(pte);
>     else
>         pr_err("vmalloc_to_page: pte_not_present for addr=%lx "
>                "PGD=%lx P4D=%lx PUD=%lx PMD=%lx PTE=%lx\n",
>                addr, pgd_val(*pgd), p4d_val(*p4d), pud_val(*pud),
>                pmd_val(*pmd), pte_val(pte));
> 
>   In arch/x86/kernel/alternative.c::__text_poke()
> 
>     /* BUG 1: pages[0] NULL */
>     if (!pages[0] || ...) {
>         pr_err("%s: vmalloc_to_page failed for addr=%pK pages[0]=%pK pages[1]=%pK cross=%d\n", ...);
>         BUG();
>     }
> 
>     ...
> 
>     /* BUG 2: memcmp mismatch */
>     if (memcmp(addr, src, len)) {
>         struct page *page_now = vmalloc_to_page(addr);
>         pr_err("%s: memcmp failed at addr=%pK len=%zu\n", __func__, addr, len);
>         pr_err("%s: expected (src): %*ph\n", __func__, (int)len, src);
>         pr_err("%s: found    (dst): %*ph\n", __func__, (int)len, (u8 *)addr);
>         pr_err("%s: pfn at poke time (vmalloc_to_page): %lx\n", __func__, page_to_pfn(pages[0]));
>         pr_err("%s: pfn at readback time (vmalloc_to_page): %lx\n", __func__, page_to_pfn(page_now));
>         text_poke_dump_pagetable((unsigned long)addr); /* reads CR3 */
>         BUG();
>     }
> 
> Signature 1 — vmalloc_to_page() returned NULL:
> 
>   vmalloc_to_page: pte_not_present for addr=ffffffffc0697979 PGD=28242067 P4D=28243067 PUD=28245067 PMD=3be001a1 PTE=0
>   SMP alternatives: __text_poke: vmalloc_to_page failed for addr=ffffffffc0697979 pages[0]=0000000000000000 pages[1]=0000000000000000 cross=0
> 
>   PMD was not a leaf, and the code reached a point to check pte_present() and
>   finds the PTE to be not present. But in the dump in else case, pmd_val(*pmd)
>   shows that it is a PMD leaf (PSE bit set).
> 
> Signature 2 — vmalloc_to_page() returned a wrong page:
> 
>   SMP alternatives: __text_poke: memcmp failed at addr=ffffffffc06fc4bd len=1
>   SMP alternatives: __text_poke: expected (src): cc
>   SMP alternatives: __text_poke: found    (dst): e8
>   SMP alternatives: __text_poke: pfn at poke time (vmalloc_to_page): 3884a
>   SMP alternatives: __text_poke: pfn at readback time (vmalloc_to_page): 52fc
>   SMP alternatives: __text_poke: page table dump for addr:
>     PGD 1ba42067  P4D 1ba43067  PUD 1ba45067  PMD 52001a1
> 
>   vmalloc_to_page() returned pfn=0x3884a when pages[0] was queried. The write
>   of 0xcc landed there. When vmalloc_to_page() was called again during the
>   memcmp readback in the instrumentation code, the vmap mapping resolved through
>   PMD=52001a1 to a different pfn=0x52fc, which contains 0xe8 (existing
>   instruction bytes). This is silent memory corruption — 0xcc was written to
>   the wrong physical page (pfn=0x3884a).
> 
> BISECTION
> ---------
> 
> The following experiments all ran 200 iterations of 512 concurrent guest boots
> with zero crashes.
> 
>   1. Remove "select ARCH_HAS_EXECMEM_ROX" from arch/x86/Kconfig
> 
>   2. Boot with "nohugevmalloc"
> 
>   3. Remove VM_ALLOW_HUGE_VMAP from vm_flags in execmem_cache_populate()
> 
>        - unsigned long vm_flags = VM_ALLOW_HUGE_VMAP;
>        + unsigned long vm_flags = 0;
> 
>   4. Remove CPA_COLLAPSE from set_memory_rox()
> 
>        return change_page_attr_set_clr(&addr, numpages, __pgprot(0), clr, 0,
>   -                                    CPA_COLLAPSE, NULL);
>   +                                    0, NULL);
> 
>      Without CPA_COLLAPSE, set_memory_rox() changes 4K PTE permissions
>      in-place and the PMD stays permanently as a PTE-page pointer after
>      the first split — no re-collapse, no future race window. This
>      confirms that CPA_COLLAPSE is a necessary part of the race sequence.
> 
>   5. Hold pgd_lock around vmalloc_to_page() in __text_poke()
> 
>          if (!core_kernel_text((unsigned long)addr)) {
>      +           spin_lock(&pgd_lock);
>                  pages[0] = vmalloc_to_page(addr);
>                  if (cross_page_boundary)
>                          pages[1] = vmalloc_to_page(addr + PAGE_SIZE);
>      +           spin_unlock(&pgd_lock);
>          } else {
> 
>      __split_large_page() holds pgd_lock while installing the new PMD
>      and populating all 512 4K PTEs. Holding pgd_lock around
>      vmalloc_to_page() prevents CPA from modifying the PMD concurrently
>      with the page table walk. This confirms the race is precisely
>      between vmalloc_to_page() reading the PMD/PTE and
>      __split_large_page() modifying it under pgd_lock.
> 
>      Note: this is a diagnostic experiment, not a production fix.
> 
> The experiments point to a race between vmalloc_to_page() in
> __text_poke() and __split_large_page() modifying the shared PMD under
> pgd_lock. __split_large_page() holds pgd_lock while modifying the
> PMD; vmalloc_to_page() holds no lock and can observe the PMD in an
> intermediate state.
> 
> What would be the correct fix for this issue?
> 
> Happy to test any proposed fix or run further experiments.

https://lore.kernel.org/linux-mm/anCK3eWFMwZqq5ka@pedro-suse/

-- 
Pedro