Re: rtentry_free panic

On Wed, Aug 20, 2025 at 02:30:20PM +0200, Kristof Provost wrote:
> Hi,
> 
> Running the pf tests I very occasional (say 1 out of 10 runs) see panics
> freeing an rtentry.
> This mostly manifests during bricoler test runs, and usually with the KMSAN
> kernel config. I assume that’s because there’s a timing factor involved
> rather than it being an issue that’s directly detected by KMSAN/KASAN.

I've seen this before, but not in the past few months.  I'm running with
the default parallelism of 4 most of the time.

> Here’s the panic:
> 
> 	Freed UMA keg (rtentry) was not empty (2 items).  Lost 1 pages of memory.
> 
> 
> 	Fatal trap 12: page fault while in kernel mode
> 	cpuid = 3; apic id = 03
> 	fault virtual address   = 0x2
> 	fault code              = supervisor read data, page not present
> 	instruction pointer     = 0x20:0xffffffff81896d53
> 	stack pointer           = 0x28:0xfffffe0098468b20
> 	frame pointer           = 0x28:0xfffffe0098468bb0
> 	code segment            = base 0x0, limit 0xfffff, type 0x1b
> 	                        = DPL 0, pres 1, long 1, def32 0, gran 1
> 	processor eflags        = interrupt enabled, resume, IOPL = 0
> 	current process         = 0 (softirq_3)
> 	rdi: 0000000000000000 rsi: fffffe00e08b67e0 rdx: 0000000000000000
> 	rcx: fffffe000c5f08d8  r8: 0000000000000000  r9: 0000000000000001
> 	rax: fffffe0000000000 rbx: 0000000000000000 rbp: fffffe0098468bb0
> 	r10: 0000000000000001 r11: 0000000000000005 r12: 0000000000000000
> 	r13: fffffe0155c46920 r14: 0000000000000000 r15: fffffe00e08b67e0
> 	trap number             = 12
> 	panic: page fault
> 	cpuid = 3
> 	time = 1754664399
> 	KDB: stack backtrace:
> 	db_trace_self_wrapper() at db_trace_self_wrapper+0xa5/frame
> 0xfffffe0098468390
> 	kdb_backtrace() at kdb_backtrace+0xc6/frame 0xfffffe00984684f0
> 	vpanic() at vpanic+0x214/frame 0xfffffe0098468690
> 	panic() at panic+0xb5/frame 0xfffffe0098468750
> 	trap_pfault() at trap_pfault+0x7e4/frame 0xfffffe0098468870
> 	trap() at trap+0x765/frame 0xfffffe0098468a50
> 	calltrap() at calltrap+0x8/frame 0xfffffe0098468a50
> 	--- trap 0xc, rip = 0xffffffff81896d53, rsp = 0xfffffe0098468b20, rbp =
> 0xfffffe0098468bb0 ---
> 	uma_zfree_arg() at uma_zfree_arg+0x23/frame 0xfffffe0098468bb0
> 	destroy_rtentry_epoch() at destroy_rtentry_epoch+0x17a/frame
> 0xfffffe0098468c70
> 	epoch_call_task() at epoch_call_task+0x26d/frame 0xfffffe0098468d50
> 	gtaskqueue_run_locked() at gtaskqueue_run_locked+0x366/frame
> 0xfffffe0098468eb0
> 	gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0x138/frame
> 0xfffffe0098468ef0
> 	fork_exit() at fork_exit+0xa3/frame 0xfffffe0098468f30
> 	fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0098468f30
> 	--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> 	KDB: enter: panic
> 	[ thread pid 0 tid 100010 ]
> 	Stopped at      kdb_enter+0x34: movq    $0,0x20d7651(%rip)
> 
> We’re panicing because the V_rtzone zone has been cleaned up (in
> vnet_rtzone_destroy()). I explicitly NULL out V_rtzone too, to make this
> more obvious.
> Note that we failed to completely free all rtentries (`Freed UMA keg
> (rtentry) was not empty (2 items).  Lost 1 pages of memory.`). Presumably at
> least on of those two gets freed later, and that’s the panic we see.
> 
> rt_free() queues the actual delete as an epoch callback
> (`NET_EPOCH_CALL(destroy_rtentry_epoch, &rt->rt_epoch_ctx);`), and that’s
> what we see here: the zone is removed before we’re done freeing all of the
> rtentries.
> 
> vnet_rtzone_destroy() is called from rtables_destroy(), but that explicitly
> calls NET_EPOCH_DRAIN_CALLBACKS() first, so I’d expect all of the pending
> cleanups to have been done at that point.  The comment block above does
> suggest that there may still be nexthop entries pending deletion even after
> the we drain the callbacks. I think I can see how that’d happen for
> nexthops, but I do not see how it can happen for rtentries.

Is it possible that if_detach_internal()->rt_flushifroutes() is running
after the rtentry zone is being destroyed?  That is, maybe we're
destroying interfaces too late in the jail teardown process?

> Has anyone else seen this panic or have any ideas what I’m missing?
> 
> Thanks,
> Kristof