Re: aarch64 main-n263493-4e8d558c9d1c-dirty (so: 2023-Jun-10) Kyuafile run: "Fatal data abort" crash during vnet_register_sysinit

On Jun 24, 2023, at 10:49, Mark Millard <[email protected]> wrote:

> On Jun 24, 2023, at 10:00, Mark Millard <[email protected]> wrote:
> 
>> The running system build is a non-debug build (but
>> with symbols not stripped).
>> 
>> The HoneyComb's console log shows:
>> 
>> . . .
>> GEOM_STRIPE: Device stripe.IMfBZr destroyed.
>> GEOM_NOP: Device md0.nop created.
>> g_vfs_done():md0.nop[READ(offset=5885952, length=8192)]error = 5
>> GEOM_NOP: Device md0.nop removed.
>> GEOM_NOP: Device md0.nop created.
>> g_vfs_done():md0.nop[READ(offset=5935104, length=4096)]error = 5
>> g_vfs_done():md0.nop[READ(offset=5935104, length=4096)]error = 5
>> GEOM_NOP: Device md0.nop removed.
>> GEOM_NOP: Device md0.nop created.
>> GEOM_NOP: Device md0.nop removed.
>> Fatal data abort:
>> x0: ffffa02506e64400
>> x1: ffff0001ea401880 (g_raid3_post_sync + 3a145f8)
>> x2:               4b
>> x3: a343932b0b22fb30
>> x4:                0
>> x5:  3310b0d062d0e1d
>> x6: 1d0e2d060d0b3103
>> x7:                0
>> x8:         ea325df8
>> x9: ffff0001eec946d0 ($d.6 + 0)
>> x10: ffff0001ea401880 (g_raid3_post_sync + 3a145f8)
>> x11:                0
>> x12:                0
>> x13: ffff000000cd8960 (lock_class_mtx_sleep + 0)
>> x14:                0
>> x15: ffffa02506e64405
>> x16: ffff0001eec94860 (_DYNAMIC + 160)
>> x17: ffff00000063a450 (ifc_attach_cloner + 0)
>> x18: ffff0001eb290400 (g_raid3_post_sync + 48a3178)
>> x19: ffff0001eec94600 (vnet_epair_init_vnet_init + 0)
>> x20: ffff000000fa5b68 (vnet_sysinit_sxlock + 18)
>> x21: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0)
>> x22: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0)
>> x23: ffffa0000042e500
>> x24: ffffa0000042e500
>> x25: ffff000000ce0788 (linker_lookup_set_desc + 0)
>> x26: ffffa0203cdef780
>> x27: ffff0001eec94698 (__set_sysinit_set_sym_if_epairmodule_sys_init + 0)
>> x28: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0)
>> x29: ffff0001eb290430 (g_raid3_post_sync + 48a31a8)
>> sp: ffff0001eb290400
>> lr: ffff0001eec82a4c ($x.1 + 3c)
>> elr: ffff0001eec82a60 ($x.1 + 50)
>> spsr:         60000045
>> far: ffff0002d8fba4c8
>> esr:         96000046
>> panic: vm_fault failed: ffff0001eec82a60 error 1
>> cpuid = 14
>> time = 1687625470
>> KDB: stack backtrace:
>> db_trace_self() at db_trace_self
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
>> vpanic() at vpanic+0x13c
>> panic() at panic+0x44
>> data_abort() at data_abort+0x2fc
>> handle_el1h_sync() at handle_el1h_sync+0x14
>> --- exception, esr 0x96000046
>> $x.1() at $x.1+0x50
>> vnet_register_sysinit() at vnet_register_sysinit+0x114
>> linker_load_module() at linker_load_module+0xae4
>> kern_kldload() at kern_kldload+0xfc
>> sys_kldload() at sys_kldload+0x60
>> do_el0_sync() at do_el0_sync+0x608
>> handle_el0_sync() at handle_el0_sync+0x44
>> --- exception, esr 0x56000000
>> KDB: enter: panic
>> [ thread pid 70419 tid 101003 ]
>> Stopped at      kdb_enter+0x44: str     xzr, [x19, #3200]
>> db> 
>> 
>> I'll see if a re-run is repeatable.
>> 
> 
> It repeats:
> 
> GEOM_STRIPE: Device stripe/stripe.VkbPk1 deactivated.
> GEOM_STRIPE: Disk md1 removed from stripe.VkbPk1.
> GEOM_STRIPE: Disk md0 removed from stripe.VkbPk1.
> GEOM_STRIPE: Device stripe.VkbPk1 destroyed.
> GEOM_NOP: Device md0.nop created.
> g_vfs_done():md0.nop[READ(offset=5885952, length=8192)]error = 5
> GEOM_NOP: Device md0.nop removed.
> GEOM_NOP: Device md0.nop created.
> g_vfs_done():md0.nop[READ(offset=5935104, length=4096)]error = 5
> g_vfs_done():md0.nop[READ(offset=5935104, length=4096)]error = 5
> GEOM_NOP: Device md0.nop removed.
> GEOM_NOP: Device md0.nop created.
> GEOM_NOP: Device md0.nop removed.
> Fatal data abort:
>  x0: ffffa0003b1a9500
>  x1: ffff00021b530260
>  x2:               4b
>  x3: a343932b0b22fb30
>  x4:                0
>  x5:  3310b0d062d0e1d
>  x6: 1d0e2d060d0b3103
>  x7:                0
>  x8:         ea325df8
>  x9: ffff00021d6946d0 ($d.6 + 0)
> x10: ffff00021b530260
> x11:                0
> x12:                0
> x13: ffff000000cd8960 (lock_class_mtx_sleep + 0)
> x14:                0
> x15: ffffa0003b1a9505
> x16: ffff00021d694860 (_DYNAMIC + 160)
> x17: ffff00000063a450 (ifc_attach_cloner + 0)
> x18: ffff00021a6ea400
> x19: ffff00021d694600 (vnet_epair_init_vnet_init + 0)
> x20: ffff000000fa5b68 (vnet_sysinit_sxlock + 18)
> x21: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0)
> x22: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0)
> x23: ffffa00000431500
> x24: ffffa00000431500
> x25: ffff000000ce0788 (linker_lookup_set_desc + 0)
> x26: ffffa02e1ab6d180
> x27: ffff00021d694698 (__set_sysinit_set_sym_if_epairmodule_sys_init + 0)
> x28: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0)
> x29: ffff00021a6ea430
>  sp: ffff00021a6ea400
>  lr: ffff00021d682a4c ($x.1 + 3c)
> elr: ffff00021d682a60 ($x.1 + 50)
> spsr:         60000045
> far: ffff0003079ba4c8
> esr:         96000046
> panic: vm_fault failed: ffff00021d682a60 error 1
> cpuid = 1
> time = 1687628622
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
> vpanic() at vpanic+0x13c
> panic() at panic+0x44
> data_abort() at data_abort+0x2fc
> handle_el1h_sync() at handle_el1h_sync+0x14
> --- exception, esr 0x96000046
> $x.1() at $x.1+0x50
> vnet_register_sysinit() at vnet_register_sysinit+0x114
> linker_load_module() at linker_load_module+0xae4
> kern_kldload() at kern_kldload+0xfc
> sys_kldload() at sys_kldload+0x60
> do_el0_sync() at do_el0_sync+0x608
> handle_el0_sync() at handle_el0_sync+0x44
> --- exception, esr 0x56000000
> KDB: enter: panic
> [ thread pid 36377 tid 100985 ]
> Stopped at      kdb_enter+0x44: str     xzr, [x19, #3200]
> db> 
> 
> 
> For reference, the output of the run in the ssh
> session ends with:
> 
> . . .
> sys/kqueue/libkqueue/kqueue_test:main  ->  passed  [48.258s]
> sys/mac/bsdextended/ugidfw_test:main  ->  skipped: mac_bsdextended not loaded  [0.006s]
> sys/mac/portacl/nobody_test:main  ->  skipped: MAC_PORTACL is unavailable.  [0.010s]
> sys/mac/portacl/root_test:main  ->  skipped: MAC_PORTACL is unavailable.  [0.010s]
> sys/mqueue/mqueue_test:mqtest1  ->  passed  [0.025s]
> sys/mqueue/mqueue_test:mqtest2  ->  passed  [0.025s]
> sys/mqueue/mqueue_test:mqtest5  ->  passed  [0.025s]
> sys/net/if_ovpn/if_ovpn_c:tcp  ->  skipped: if_ovpn not loaded  [0.006s]
> sys/netinet/arp:arp_add_success  ->  
> 
> That should give some extra information about the context
> of failure.

So I installed, booted, and tried my debug build. It failed
the same way in the same place, with no extra console
reporting for the crash by the debug code: no assertion
failures or WITNESS reports or the like first.


===
Mark Millard
marklmi at yahoo.com