Re: aarch64 main [so: 15] panic's in kyua's sys/net/if_lagg_test:status_stress [confirmed with snapshot kernel]

On Sep 11, 2023, at 01:13, Mark Millard <[email protected]> wrote:

> It will be some time before I can try this with
> an official snapshot instead of a personal build.
> The build is based on b6ce41118bb1 :
> 
> # uname -apKU
> FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #17 main-n265279-b6ce41118bb1-dirty: Sun Sep 10 14:36:47 PDT 2023     root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000
> 
> So it was a non-debug build, although I do not
> strip symbols and such in my builds.
> 
> . . .
> sys/net/if_lagg_test:create  ->  passed  [0.105s]
> sys/net/if_lagg_test:create_destroy_stress  ->  skipped: Skipping this test because it easily panics the machine  [0.019s]
> sys/net/if_lagg_test:lacp_linkstate_destroy_stress  ->  passed  [60.045s]
> sys/net/if_lagg_test:set_ether  ->  passed  [0.066s]
> sys/net/if_lagg_test:status_stress  ->  
> 
> The core.txt.5 is not great, unfortunately:
> 
> panic: vm_fault failed: 0xffff0000006b96dc error 1
> 
> GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD]
> . . .
> Reading symbols from /boot/kernel/kernel...
> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
> 
> Unread portion of the kernel message buffer:
> (dump_iface + 0x2c0)
> elr: 0xffff0000006b96dc (dump_sa + 0x1c)
> spsr: 0x0000000000400045
> far: 0x44572d4338374144
> esr: 0x0000000096000004
> panic: vm_fault failed: 0xffff0000006b96dc error 1
> cpuid = 2
> time = 1694414226
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
> vpanic() at vpanic+0x1a0
> panic() at panic+0x44
> data_abort() at data_abort+0x304
> handle_el1h_sync() at handle_el1h_sync+0x14
> --- exception, esr 0x96000004
> dump_sa() at dump_sa+0x1c
> dump_iface() at dump_iface+0x2bc
> dump_cb() at dump_cb+0x18
> if_foreach_sleep() at if_foreach_sleep+0x244
> rtnl_handle_getlink() at rtnl_handle_getlink+0xec
> rtnl_handle_message() at rtnl_handle_message+0x19c
> nl_taskqueue_handler() at nl_taskqueue_handler+0x674
> taskqueue_run_locked() at taskqueue_run_locked+0x194
> taskqueue_thread_loop() at taskqueue_thread_loop+0xcc
> fork_exit() at fork_exit+0x88
> fork_trampoline() at fork_trampoline+0x14
> KDB: enter: panic
> 
> get_curthread () at /usr/main-src/sys/arm64/include/pcpu.h:77
> 77              __asm __volatile("ldr   %0, [x18]" : "=&r"(td));
> (kgdb) #0  get_curthread () at /usr/main-src/sys/arm64/include/pcpu.h:77
> #1  doadump (textdump=0, textdump@entry=4003518992)
>    at /usr/main-src/sys/kern/kern_shutdown.c:405
> #2  0xffff0000000f7704 in db_dump (dummy=<optimized out>,      dummy2=<optimized out>, dummy3=<optimized out>, dummy4=<optimized out>)
>    at /usr/main-src/sys/ddb/db_command.c:591
> #3  0xffff0000000f74e0 in db_command (last_cmdp=<optimized out>,      cmd_table=<optimized out>, dopager=true)
>    at /usr/main-src/sys/ddb/db_command.c:504
> #4  0xffff0000000f71b8 in db_command_loop ()
>    at /usr/main-src/sys/ddb/db_command.c:551
> #5  0xffff0000000fad9c in db_trap (type=<optimized out>, code=<optimized out>)
>    at /usr/main-src/sys/ddb/db_main.c:268
> #6  0xffff0000004f4ec4 in kdb_trap (type=60, code=0, tf=<optimized out>)
>    at /usr/main-src/sys/kern/subr_kdb.c:790
> #7  <signal handler called>
> #8  <signal handler called>
> #9  <signal handler called>
> #10 <signal handler called>
> #11 <signal handler called>
> #12 <signal handler called>
> #13 <signal handler called>
> #14 <signal handler called>
> #15 <signal handler called>
> #16 <signal handler called>
> #17 <signal handler called>
> #18 <signal handler called>
> #19 <signal handler called>
> #20 <signal handler called>
> #21 <signal handler called>
> #22 <signal handler called>
> Backtrace stopped: Cannot access memory at address 0x10
> (kgdb) 
> 
> 
> So some transcribing of a picture in order to
> show register values that were reported:
> 
> Fatal data abort:
>    x0: 0xffff000leea0e7f0 (_DYNAMIC * 0x6d816648)
>    x1: 0x0000000000000001
>    x2: 0x44572d4338374143
>    x3: 0xffff0000005d3f90 (ifdead_ioctl + 0x0)
>    x4: 0xffffa00b7f0d185e
>    x5: 0xffffa0023fe4b992
>    x6: 0x000000006767616c
>    x7: 0x00706174016f7575
>    x8: 0x00000000000001a4
>    x9: 0x0000000000210005
>   x10: 0×0000000000000800
>   x11: 0xfefefefefefefeff
>   x12: 0x0000000000000008
>   x13: 0x0000000000000000
>   x14: 0x00000000000000ff
>   x15: 0x0000000000000700
>   x16: 0x0000000000000008
>   x17: 0x0000000000000007
>   x18: 0xffff0001eea0e500 (_DYNAMIC + 0x6d816358)
>   x19: 0xffff000leea0e7f0 (_DYNAMIC * 0x6d816648)
>   x20: 0xffffa00b7f0d1800
>   x21: 0xffffa00b7f0d1858
>   x22: 0x000000000000000c
>   x23: 0X0000000000000005
>   x24: 0×0000000000000000
>   x25: 0xffff000000c68000 (sysctl___kern_features_netlink + 0x10)
>   x26: 0x0000000000000000
>   x27: 0xffff000000ce9000 (cap_linkat_source_rights + 0x8)
>   x28: 0xffff0000006bb0a0 (dump_cb + 0x0)
>   x29: 0xffff0001eea0e520 (_DYNAMIC + 0x6d816378)
>    sp: 0xffff0001eea0e500
>    lr: 0xffff0000006b8fe0 (dump_iface + 0x2c0)
>   elr: 0xffff0000006b96dc (dump_sa + 0x1c)
>  spsr: 0x0000000000400045
>   far: 0x44572d4338374144
>   esr: 0x0000000096000004
> panic: m_fault failed: 0xffff0000006b96dc error 1
> 
> I expect that this is similar to reports I'd made
> back in 14.0-CURRENT days. As I remember, snapshot
> builds of the time also got the panic.
> 
> I will note that an earlier 14.0-BETA1 snapshot
> kernel test run did not panic at this point in the
> sequence (or at any point). But I do not know how
> repeatable the panics are in the various contexts.
> 
> I'll note that I've tried to have the various ports
> installed (poudriere built) that are listed at:
> 
> https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test_image-head.sh#L69-L84
> 
> (The ones that build for aarch64, anyway.)
> 
> I had in /etc/kyua/kyua.conf :
> 
> test_suites.FreeBSD.disks = '/dev/md0 /dev/md1 /dev/md2 /dev/md3 /dev/md4 /dev/md5'
> 
> and used:
> 
> # more ~/prekyua-aarch64-mdconfig.sh 
> #! /bin/sh
> truncate -s 4g /var/tmp/for-md0.dat
> truncate -s 4g /var/tmp/for-md1.dat
> truncate -s 4g /var/tmp/for-md2.dat
> truncate -s 4g /var/tmp/for-md3.dat
> truncate -s 4g /var/tmp/for-md4.dat
> truncate -s 4g /var/tmp/for-md5.dat
> mdconfig -f /var/tmp/for-md0.dat -u md0
> mdconfig -f /var/tmp/for-md1.dat -u md1
> mdconfig -f /var/tmp/for-md2.dat -u md2
> mdconfig -f /var/tmp/for-md3.dat -u md3
> mdconfig -f /var/tmp/for-md4.dat -u md4
> mdconfig -f /var/tmp/for-md5.dat -u md5
> 
> I also did a:
> 
> # kldload linux64
> 
> before doing:
> 
> # /usr/bin/kyua test -k /usr/tests/Kyuafile
> 
> (Not true of linux64.ko in 14.0-CURRENT days.)

# uname -apKU
FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #0 main-n265205-03a7c36ddbc0: Thu Sep  7 03:05:31 UTC 2023     [email protected]:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 aarch64 1500000 1500000

# /usr/bin/kyua test -k /usr/tests/Kyuafile sys/net/if_lagg_test:status_stress
sys/net/if_lagg_test:status_stress  ->  

got:

panic: vm_fault failed: 0xffff0000006813b4 error 1

GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD]
. . .
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
<6>ue0: 3 link states coalesced
<6>ue0: link state changed to UP
<6>lagg0: link state changed to DOWN
<6>ue0: link state changed to DOWN
Fatal data abort:
  x0: 0xffff00015df8d800 (infiniband_input.printedonce + 0x11eff68)
  x1: 0x0000000000000001
  x2: 0xdeadc0dedeadc0de
  x3: 0xffff000000593e34 (ifdead_ioctl + 0x0)
  x4: 0xffffa0004fb6285e
  x5: 0xffffa0004fc00192
  x6: 0x000000006767616c
  x7: 0x6e6d760070617401
  x8: 0x00000000000001a4
  x9: 0xffffa0004fc00000
 x10: 0x0000000000210005
 x11: 0x000000007ffffffe
 x12: 0x0000000000000008
 x13: 0x0000000000000000
 x14: 0x0000000000010000
 x15: 0x0000000000000001
 x16: 0x0000000000010000
 x17: 0x0000000000000007
 x18: 0xffff00015df8d500
<6>ue0: link state changed to UP
 (infiniband_input.printedonce + 0x11efc68)
 x19: 0xffff00015df8d800 (infiniband_input.printedonce + 0x11eff68)
 x20: 0xffffa0004fb62800
 x21: 0xffffa0004fb62858
 x22: 0x000000000000000c
 x23: 0x0000000000000005
 x24: 0x0000000000000000
 x25: 0xffff000000c58000 (sysctl___net_netlink_debug + 0x40)
 x26: 0x0000000000000000
 x27: 0xffff000000cd9000 (sdt_vfs_vop_vop_spare5_return + 0x10)
 x28: 0xffff000000cd9000 (sdt_vfs_vop_vop_spare5_return + 0x10)
 x29: 0xffff00015df8d520 (infiniband_input.printedonce + 0x11efc88)
  sp: 0xffff00015df8d500
  lr: 0xffff000000680cbc (dump_iface + 0x2c0)
 elr: 0xffff0000006813b4 (dump_sa + 0x1c)
spsr: 0x0000000000400045
 far: 0xdeadc0dedeadc0df
 esr: 0x0000000096000004
panic: vm_fault failed: 0xffff0000006813b4 error 1
cpuid = 3
time = 1694485392
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x19c
panic() at panic+0x44
data_abort() at data_abort+0x35c
handle_el1h_sync() at handle_el1h_sync+0x14
--- exception, esr 0x96000004
dump_sa() at dump_sa+0x1c
dump_iface() at dump_iface+0x2bc
dump_cb() at dump_cb+0x18
if_foreach_sleep() at if_foreach_sleep+0x254
rtnl_handle_getlink() at rtnl_handle_getlink+0xec
rtnl_handle_message() at rtnl_handle_message+0x19c
nl_taskqueue_handler() at nl_taskqueue_handler+0x5dc
taskqueue_run_locked() at taskqueue_run_locked+0x17c
taskqueue_thread_loop() at taskqueue_thread_loop+0xc8
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic

get_curthread () at /usr/src/sys/arm64/include/pcpu.h:77
77              __asm __volatile("ldr   %0, [x18]" : "=&r"(td));
(kgdb) #0  get_curthread () at /usr/src/sys/arm64/include/pcpu.h:77
#1  doadump (textdump=0, textdump@entry=1576585744)
    at /usr/src/sys/kern/kern_shutdown.c:405
#2  0xffff0000000ec18c in db_dump (dummy=<optimized out>,      dummy2=<optimized out>, dummy3=<optimized out>, dummy4=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:591
#3  0xffff0000000ebf88 in db_command (last_cmdp=<optimized out>,      cmd_table=<optimized out>, dopager=true)
    at /usr/src/sys/ddb/db_command.c:504
#4  0xffff0000000ebc80 in db_command_loop ()
    at /usr/src/sys/ddb/db_command.c:551
#5  0xffff0000000ef440 in db_trap (type=<optimized out>, code=<optimized out>)
    at /usr/src/sys/ddb/db_main.c:268
#6  0xffff0000004b4860 in kdb_trap (type=60, code=0, tf=<optimized out>)
    at /usr/src/sys/kern/subr_kdb.c:790
#7  <signal handler called>
#8  <signal handler called>
#9  <signal handler called>
#10 <signal handler called>
#11 <signal handler called>
#12 <signal handler called>
#13 <signal handler called>
#14 <signal handler called>
#15 <signal handler called>
#16 <signal handler called>
#17 <signal handler called>
#18 <signal handler called>
#19 <signal handler called>
#20 <signal handler called>
#21 <signal handler called>
#22 <signal handler called>
#23 <signal handler called>
Backtrace stopped: Cannot access memory at address 0x10
(kgdb) 

(Again, kgdb's stack frames #7 and larger are not particularly
useful.)

Possibly interesting are the slightly different values:

  x2: 0xdeadc0dedeadc0de
and:
 far: 0xdeadc0dedeadc0df


===
Mark Millard
marklmi at yahoo.com