Re: Are we running libpayload unit tests in the CI?

Great, thanks a lot for tracking this down and getting it fixed!

On Sun, Mar 3, 2024 at 12:50 PM Martin Roth <[email protected]> wrote:
>
> Hi Julius, I've fixed the issue you were seeing. Thanks for bringing it to our attention.
>
> There were 2 things that were contributing to jenkins not seeing the libpayload files:
> 1) They were deeper in the tree than we were looking for junit*.xml files. These were only getting picked up two levels deep into the payloads directory: payloads/*/*/junit*.xml
> 2) We were also looking for junit.xml files anywhere in the tree, but the output files only match junit*.xml files.
>
> Here's the list of files we needed to find:
> ./payloads/libpayload/build/coverage/junit-libpayload-tests-build.xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-has-sha512-test(tests).xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-no-sha512-test(tests).xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-has-sha512-test(tests).xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-no-sha512-test(tests).xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-has-fallback-test(tests).xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-no-fallback-test(tests).xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_libc_fmap_locate_area-test(tests).xml
> ./payloads/libpayload/build/coverage/junit-libpayload-tests_drivers_speaker-test(tests).xml
>
> I've updated the general pattern to find junit*.xml files anywhere in the tree, and now they're seen by the jenkins log collector.
>
> In the output of https://qa.coreboot.org/job/coreboot-gerrit/253059/ you can see that we now get 4 errors:
> Test Result (4 failures / +4)
> tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned
> tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned
> tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned
> tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned
>
> Thanks again for helping to fix this problem.
>
> Martin
>
> Feb 28, 2024, 16:22 by [email protected]:
>
> > Hmm... well, I just uploaded a CL that clearly breaks one of the
> > tests, and Jenkins still gave me a Verified+1:
> > https://review.coreboot.org/80788
> >
> > I also don't see any info about the tests running on the test report
> > (https://qa.coreboot.org/job/coreboot-gerrit/252941/testReport/(root)/).
> > So if the tests are running (and the code you pointed out strongly
> > looks like they should be), it's probably a problem with reading back
> > the test results? How does Jenkins collect those XML files?
> >
> > On Tue, Feb 27, 2024 at 7:04 PM Martin Roth <[email protected]> wrote:
> >
> >>
> >> Hey Julius,
> >>  Here's what we running:
> >> https://review.coreboot.org/plugins/gitiles/coreboot/+/refs/heads/main/util/testing/Makefile.mk#106
> >>
> >> Martin
> >> Feb 27, 2024 at 18:31 by [email protected]:
> >>
> >> > Hi,
> >> >
> >> > We've had unit test support in libpayload for over two years now, but
> >> > I just noticed that it doesn't look like we're actually running them
> >> > in the CI? E.g. if I look through a test run like this
> >> > https://qa.coreboot.org/job/coreboot-gerrit/252761/testReport/(root)/
> >> > I can't find them anywhere (most of them have confusingly similar
> >> > names with coreboot unit tests, but "fmap_locate_area-test" is a
> >> > unique one).
> >> >
> >> > Is that intended or an oversight? Would it be easy to fix? (I'm not
> >> > super familiar with how the tests are integrated into Jenkins, but the
> >> > libpayload framework is pretty much a direct copy of the coreboot
> >> > framework, so if we can run the coreboot ones hopefully it should be
> >> > easy to run the libpayload ones as well?)
> >> > _______________________________________________
> >> > coreboot mailing list -- [email protected]
> >> > To unsubscribe send an email to [email protected]
> >> >
> >>
> > _______________________________________________
> > coreboot mailing list -- [email protected]
> > To unsubscribe send an email to [email protected]
> >
>
_______________________________________________
coreboot mailing list -- [email protected]
To unsubscribe send an email to [email protected]