Re: FreeBSD-stable-15-amd64-test - Build #30 - Still Failing

Given I cannot reproduce this problem and given that Jenkins is still 
having an issue with this test, would it be possible to obtain a dump from 
Jenkins? At this point the only data structure in play is the mbuf. Either 
the mbuf has disappeared or a pointer within the mbuf points to an invalid 
address.

Up to now we're just guessing.


-- 
Cheers,
Cy Schubert <[email protected]>
FreeBSD UNIX:  <[email protected]>   Web:  https://FreeBSD.org
NTP:           <[email protected]>    Web:  https://nwtime.org

			e**(i*pi)+1=0



In message <[email protected]>, Cy Schubert writes:
> The tests have been run on a couple of my 16-CURRENT machines. No panics 
> were experienced. I need more information. Though, my guess is that pf or 
> the stack may have freed the mbuf before ipfilter managed to finish 
> inspecting it.
>
>
> -- 
> Cheers,
> Cy Schubert <[email protected]>
> FreeBSD UNIX:  <[email protected]>   Web:  https://FreeBSD.org
> NTP:           <[email protected]>    Web:  https://nwtime.org
>
> 			e**(i*pi)+1=0
>
>
> In message <[email protected]>, Cy Schubert writes:
> > Thanks. I'll try the pf tests on my sandbox machine here.
> >
> >
> > -- 
> > Cheers,
> > Cy Schubert <[email protected]>
> > FreeBSD UNIX:  <[email protected]>   Web:  https://FreeBSD.org
> > NTP:           <[email protected]>    Web:  https://nwtime.org
> >
> > 			e**(i*pi)+1=0
> >
> >
> > In message <[email protected]>, Lexi Winter writes:
> > > 
> > >
> > > --5tActHfoHppOuWl4
> > > Content-Type: text/plain; charset=utf-8
> > > Content-Disposition: inline
> > > Content-Transfer-Encoding: quoted-printable
> > >
> > > Rick Macklem wrote in <CAM5tNy7kS3O=3DsAsVPev+cDmOGoe1C55nPz13Z0511xdPryQ
> KF
> > =
> > > [email protected]>:
> > > > Just in case no one in networking is seeing this,
> > > > I'm reposting it here.
> > > =20
> > > cc: cy for ipfilter.
> > >
> > > > On Tue, Oct 7, 2025 at 10:44=E2=80=AFPM <[email protected]> wro
> te
> > :
> > > > >
> > > > > CAUTION: This email originated from outside of the University of Guel
> ph
> > =
> > > =2E Do not click links or open attachments unless you recognize the sende
> r 
> > =
> > > and know the content is safe. If in doubt, forward suspicious emails to I
> Th
> > =
> > > [email protected].
> > > > >
> > > > > FreeBSD-stable-15-amd64-test - Build #30 (23fbea8cf2f360dd45cb69c98f6
> f2
> > =
> > > 931fff3721e) - Still Failing
> > > > >
> > > > > Build information: https://ci.FreeBSD.org/job/FreeBSD-stable-15-amd64
> -t
> > =
> > > est/30/
> > > > > Full change log: https://ci.FreeBSD.org/job/FreeBSD-stable-15-amd64-t
> es
> > =
> > > t/30/changes
> > > > > Full build log: https://ci.FreeBSD.org/job/FreeBSD-stable-15-amd64-te
> st
> > =
> > > /30/console
> > > > >
> > > > > Status explanation:
> > > > > "Failure" - the build is suspected being broken by the following chan
> ge
> > s
> > > > > "Still Failing" - the build has not been fixed by the following chang
> es
> > =
> > >  and
> > > > >                   this is a notification to note that these changes h
> av
> > e
> > > > >                   not been fully tested by the CI system
> > > > >
> > > > > Change summaries:
> > > > > (Those commits are likely but not certainly responsible)
> > > > >
> > > > > 23fbea8cf2f360dd45cb69c98f6f2931fff3721e by rmacklem:
> > > > > kadmin.8: Document the new dump -f flag
> > > > >
> > > > >
> > > > >
> > > > > The end of the build log:
> > > > >
> > > > > [...truncated 3.96 MiB...]
> > > > > epair0b: Ethernet address: d2:f4:4c:7c:25:d3
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: c6:fa:71:45:1f:b3
> > > > > epair1b: Ethernet address: 3a:02:db:62:eb:5a
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 13
> > =
> > > 6s]
> > > > > sys/netpfil/pf/nat64:tcp_in_floating  ->  epair0a: Ethernet address: 
> b2
> > =
> > > :20:57:6c:d4:c5
> > > > > epair0b: Ethernet address: 5a:f0:ad:fb:8a:50
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 52:58:0c:ea:9b:f3
> > > > > epair1b: Ethernet address: ca:3f:da:2a:4d:86
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 12
> > =
> > > 5s]
> > > > > sys/netpfil/pf/nat64:tcp_in_if_bound  ->  epair0a: Ethernet address: 
> 52
> > =
> > > :7d:e8:24:9b:38
> > > > > epair0b: Ethernet address: 6e:8e:70:d3:10:60
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 32:25:e3:7f:fe:7c
> > > > > epair1b: Ethernet address: e6:f1:01:c1:4f:84
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 12
> > =
> > > 6s]
> > > > > sys/netpfil/pf/nat64:tcp_out_floating  ->  epair0a: Ethernet address:
>  c
> > =
> > > e:73:6d:57:a2:67
> > > > > epair0b: Ethernet address: 2a:28:48:af:e2:4a
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 82:12:d3:13:44:bb
> > > > > epair1b: Ethernet address: c2:16:12:72:31:45
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 12
> > =
> > > 7s]
> > > > > sys/netpfil/pf/nat64:tcp_out_if_bound  ->  epair0a: Ethernet address:
>  7
> > =
> > > 6:e6:ab:94:3a:a3
> > > > > epair0b: Ethernet address: 12:ae:f5:b6:b8:19
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 56:3a:12:2a:60:8a
> > > > > epair1b: Ethernet address: 76:ac:dc:80:e3:72
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 12
> > =
> > > 6s]
> > > > > sys/netpfil/pf/nat64:tos  ->  epair0a: Ethernet address: 62:39:ae:b2:
> 2a
> > =
> > > :db
> > > > > epair0b: Ethernet address: 36:9f:e7:31:94:32
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 42:37:24:71:41:ed
> > > > > epair1b: Ethernet address: d2:ca:da:c1:6c:54
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 12
> > =
> > > 6s]
> > > > > sys/netpfil/pf/nat64:udp_in  ->  epair0a: Ethernet address: 16:99:da:
> 0e
> > =
> > > :91:0d
> > > > > epair0b: Ethernet address: 3e:57:7e:63:8d:87
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 6e:95:d1:aa:b3:a9
> > > > > epair1b: Ethernet address: ba:59:41:61:6b:19
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 13
> > =
> > > 5s]
> > > > > sys/netpfil/pf/nat64:udp_out  ->  epair0a: Ethernet address: 7a:b4:11
> :1
> > =
> > > 9:87:66
> > > > > epair0b: Ethernet address: 5e:70:01:e2:ea:b6
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 52:fd:81:7b:fc:61
> > > > > epair1b: Ethernet address: 1e:ba:76:e2:79:36
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair1b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 12
> > =
> > > 7s]
> > > > > sys/netpfil/pf/nat64:v6_gateway  ->  epair0a: Ethernet address: ee:55
> :d
> > =
> > > 2:9b:91:07
> > > > > epair0b: Ethernet address: fa:df:53:46:63:90
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 1e:a2:5a:f3:ee:cf
> > > > > epair1b: Ethernet address: 26:8a:99:45:02:97
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair2a: Ethernet address: ae:b7:eb:ed:ec:d8
> > > > > epair2b: Ethernet address: 5a:ef:f6:5c:17:bd
> > > > > epair2a: link state changed to UP
> > > > > epair2b: link state changed to UP
> > > > > epair2b: link state changed to DOWN
> > > > > epair2a: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair1b: link state changed to DOWN
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > failed: atf-check failed; see the output of the test for details  [0.
> 19
> > =
> > > 7s]
> > > > > sys/netpfil/pf/nat64.py:TestNAT64::test_address_unreachable  ->  epai
> r0
> > =
> > > a: Ethernet address: f6:fd:4b:c1:53:2d
> > > > > epair0b: Ethernet address: 5e:59:15:d0:79:82
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 02:1f:0f:68:f7:17
> > > > > epair1b: Ethernet address: 92:43:e4:76:55:96
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair2a: Ethernet address: 7a:d1:1e:45:06:b3
> > > > > epair2b: Ethernet address: 4a:7d:09:c7:93:73
> > > > > epair2a: link state changed to UP
> > > > > epair2b: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: promiscuous mode enabled
> > > > > lo0: promiscuous mode disabled
> > > > > epair0a: link state changed to DOWN
> > > > > epair0b: link state changed to DOWN
> > > > > epair2a: link state changed to DOWN
> > > > > epair2b: link state changed to DOWN
> > > > > epair1a: link state changed to DOWN
> > > > > epair1b: link state changed to DOWN
> > > > > failed: /usr/tests/sys/netpfil/pf/nat64.py:173: AttributeError  [4.71
> 0s
> > ]
> > > > > sys/netpfil/pf/nat64.py:TestNAT64::test_bad_len  ->  epair0a: Etherne
> t 
> > =
> > > address: 2e:b0:05:ae:d6:d8
> > > > > epair0b: Ethernet address: 16:e7:ed:1d:32:aa
> > > > > epair0a: link state changed to UP
> > > > > epair0b: link state changed to UP
> > > > > epair1a: Ethernet address: 8e:fb:19:37:fb:8e
> > > > > epair1b: Ethernet address: 06:53:47:b9:ae:a8
> > > > > epair1a: link state changed to UP
> > > > > epair1b: link state changed to UP
> > > > > epair2a: Ethernet address: 2a:90:ff:29:ca:a0
> > > > > epair2b: Ethernet address: 56:89:04:58:36:84
> > > > > epair2a: link state changed to UP
> > > > > epair2b: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: link state changed to UP
> > > > > lo0: promiscuous mode enabled
> > > > >
> > > > >
> > > > > Fatal trap 12: page fault while in kernel mode
> > > > > cpuid =3D 0; apic id =3D 00
> > > > > fault virtual address   =3D 0x18
> > > > > fault code              =3D supervisor read data, page not present
> > > > > instruction pointer     =3D 0x20:0xffffffff80c0f132
> > > > > stack pointer           =3D 0x28:0xfffffe008aef2980
> > > > > frame pointer           =3D 0x28:0xfffffe008aef29c0
> > > > > code segment            =3D base 0x0, limit 0xfffff, type 0x1b
> > > > >                         =3D DPL 0, pres 1, long 1, def32 0, gran 1
> > > > > processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
> > > > > current process         =3D 12 (swi1: netisr 0)
> > > > > rdi: fffffe008aef29d8 rsi: fffff8008be43e96 rdx: 00000000ffffffff
> > > > > rcx: 0000000000008466  r8: fffffe008aef2900  r9: 0000000000000003
> > > > > rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe008aef29c0
> > > > > r10: 0000000000000001 r11: 0000000000000000 r12: 0000000000000000
> > > > > r13: 000000000000000b r14: 000000000000000b r15: 00000000000001f5
> > > > > trap number             =3D 12
> > > > > panic: page fault
> > > > > cpuid =3D 0
> > > > > time =3D 1759902263
> > > > > KDB: stack backtrace:
> > > > > #0 0xffffffff80bc033d at kdb_backtrace+0x5d
> > > > > #1 0xffffffff80b73726 at vpanic+0x136
> > > > > #2 0xffffffff80b735e3 at panic+0x43
> > > > > #3 0xffffffff8107bf69 at trap_pfault+0x3c9
> > > > > #4 0xffffffff81051a88 at calltrap+0x8
> > > > > #5 0xffffffff80d84b19 at in6_cksum_partial_l2+0x109
> > > > > #6 0xffffffff8295cab5 at fr_cksum+0xa5
> > > > > #7 0xffffffff829621dd at ipf_checkl4sum+0xbd
> > > > > #8 0xffffffff829586f6 at ipf_checkv6sum+0x56
> > > > > #9 0xffffffff8295a601 at ipf_makefrip+0x1691
> > > > > #10 0xffffffff8295b6de at ipf_check+0x15e
> > > > > #11 0xffffffff82958c28 at ipf_check_wrapper6+0x58
> > > > > #12 0xffffffff80cc3438 at pfil_mbuf_in+0x38
> > > > > #13 0xffffffff80d92b94 at ip6_input+0x484
> > > > > #14 0xffffffff80cc0c98 at swi_net+0x128
> > > > > #15 0xffffffff80b2d359 at ithread_loop+0x239
> > > > > #16 0xffffffff80b298eb at fork_exit+0x7b
> > > > > #17 0xffffffff81052aae at fork_trampoline+0xe
> > > > > Uptime: 1h18m43s
> > > > > Automatic reboot in 15 seconds - press a key on the console to abort
> > > > > + rc=3D0
> > > > > + echo 'bhyve return code =3D 0'
> > > > > bhyve return code =3D 0
> > > > > + sudo /usr/sbin/bhyvectl '--vm=3Dtestvm-15-amd64-30' --destroy
> > > > > + sh -ex freebsd-ci/scripts/test/extract-meta.sh
> > > > > + METAOUTDIR=3Dmeta-out
> > > > > + rm -fr meta-out
> > > > > + mkdir meta-out
> > > > > + tar xvf meta.tar -C meta-out
> > > > > x ./
> > > > > x ./disable-zfs-tests.sh
> > > > > x ./disable-dtrace-tests.sh
> > > > > x ./auto-shutdown
> > > > > x ./run.sh
> > > > > x ./run-kyua.sh
> > > > > x ./disable-notyet-tests.sh
> > > > > + rm -f test-report.txt test-report.xml
> > > > > + mv 'meta-out/test-report.*' .
> > > > > mv: rename meta-out/test-report.* to ./test-report.*: No such file or
>  d
> > =
> > > irectory
> > > > > + report=3Dtest-report.xml
> > > > > + [ -e freebsd-ci/jobs/FreeBSD-stable-15-amd64-test/xfail-list -a -e 
> te
> > =
> > > st-report.xml ]
> > > > > + rm -f disk-cam
> > > > > + jot 5
> > > > > + rm -f disk1
> > > > > + rm -f disk2
> > > > > + rm -f disk3
> > > > > + rm -f disk4
> > > > > + rm -f disk5
> > > > > + rm -f disk-test.img
> > > > > [PostBuildScript] - [INFO] Executing post build scripts.
> > > > > [FreeBSD-stable-15-amd64-test] $ /bin/sh -xe /tmp/jenkins121969929221
> 24
> > =
> > > 583673.sh
> > > > > + ./freebsd-ci/artifact/post-link.py
> > > > > Post link: {'job_name': 'FreeBSD-stable-15-amd64-test', 'commit': '23
> fb
> > =
> > > ea8cf2f360dd45cb69c98f6f2931fff3721e', 'branch': 'stable-15', 'target': '
> am
> > =
> > > d64', 'target_arch': 'amd64', 'link_type': 'latest_tested'}
> > > > > "Link created: stable-15/latest_tested/amd64/amd64 -> ../../23fbea8cf
> 2f
> > =
> > > 360dd45cb69c98f6f2931fff3721e/amd64/amd64\n"
> > > > > Recording test results
> > > > > ERROR: Step =E2=80=98Publish JUnit test result report=E2=80=99 failed
> : 
> > =
> > > No test report files were found. Configuration error?
> > > > > Checking for post-build
> > > > > Performing post-build step
> > > > > Checking if email needs to be generated
> > > > > Email was triggered for: Failure - Any
> > > > > Sending email for trigger: Failure - Any
> > > > > Sending mail from default account using System Admin e-mail address
> > > >=20
> > >
> > > --5tActHfoHppOuWl4
> > > Content-Type: application/pgp-signature; name=signature.asc
> > >
> > > -----BEGIN PGP SIGNATURE-----
> > >
> > > iHUEABYKAB0WIQSyjTg96lp3RifySyn1nT63mIK/YAUCaOZilAAKCRD1nT63mIK/
> > > YLi5AQD7wS2pcYg+ig9nhZmwuklZlqnpXyi/Hylmv/rYUPccpgD+JCXJLVTFlH02
> > > Y4I64B5dilhI+fS+2aiv3A07ZJGXbA4=
> > > =kxko
> > > -----END PGP SIGNATURE-----
> > >
> > > --5tActHfoHppOuWl4--
> >
>