Re: Are we running libpayload unit tests in the CI?

Hi Julius, I've fixed the issue you were seeing. Thanks for bringing it to our attention.

There were 2 things that were contributing to jenkins not seeing the libpayload files:
1) They were deeper in the tree than we were looking for junit*.xml files. These were only getting picked up two levels deep into the payloads directory: payloads/*/*/junit*.xml
2) We were also looking for junit.xml files anywhere in the tree, but the output files only match junit*.xml files.

Here's the list of files we needed to find:
./payloads/libpayload/build/coverage/junit-libpayload-tests-build.xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-has-sha512-test(tests).xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-no-verification-no-sha512-test(tests).xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-has-sha512-test(tests).xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-verification-no-sha512-test(tests).xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-has-fallback-test(tests).xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_libcbfs_cbfs-lookup-no-fallback-test(tests).xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_libc_fmap_locate_area-test(tests).xml
./payloads/libpayload/build/coverage/junit-libpayload-tests_drivers_speaker-test(tests).xml

I've updated the general pattern to find junit*.xml files anywhere in the tree, and now they're seen by the jenkins log collector.

In the output of https://qa.coreboot.org/job/coreboot-gerrit/253059/ you can see that we now get 4 errors:
Test Result (4 failures / +4)
tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned
tests_libcbfs_cbfs-lookup-has-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned
tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, aligned
tests_libcbfs_cbfs-lookup-no-fallback-test(tests).test_cbfs_unverified_area_map, RW, unaligned

Thanks again for helping to fix this problem.

Martin

Feb 28, 2024, 16:22 by [email protected]:

> Hmm... well, I just uploaded a CL that clearly breaks one of the
> tests, and Jenkins still gave me a Verified+1:
> https://review.coreboot.org/80788
>
> I also don't see any info about the tests running on the test report
> (https://qa.coreboot.org/job/coreboot-gerrit/252941/testReport/(root)/).
> So if the tests are running (and the code you pointed out strongly
> looks like they should be), it's probably a problem with reading back
> the test results? How does Jenkins collect those XML files?
>
> On Tue, Feb 27, 2024 at 7:04 PM Martin Roth <[email protected]> wrote:
>
>>
>> Hey Julius,
>>  Here's what we running:
>> https://review.coreboot.org/plugins/gitiles/coreboot/+/refs/heads/main/util/testing/Makefile.mk#106
>>
>> Martin
>> Feb 27, 2024 at 18:31 by [email protected]:
>>
>> > Hi,
>> >
>> > We've had unit test support in libpayload for over two years now, but
>> > I just noticed that it doesn't look like we're actually running them
>> > in the CI? E.g. if I look through a test run like this
>> > https://qa.coreboot.org/job/coreboot-gerrit/252761/testReport/(root)/
>> > I can't find them anywhere (most of them have confusingly similar
>> > names with coreboot unit tests, but "fmap_locate_area-test" is a
>> > unique one).
>> >
>> > Is that intended or an oversight? Would it be easy to fix? (I'm not
>> > super familiar with how the tests are integrated into Jenkins, but the
>> > libpayload framework is pretty much a direct copy of the coreboot
>> > framework, so if we can run the coreboot ones hopefully it should be
>> > easy to run the libpayload ones as well?)
>> > _______________________________________________
>> > coreboot mailing list -- [email protected]
>> > To unsubscribe send an email to [email protected]
>> >
>>
> _______________________________________________
> coreboot mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>

_______________________________________________
coreboot mailing list -- [email protected]
To unsubscribe send an email to [email protected]