Re: [PATCH 5/6] selftests: Preserve subtarget failures in all/install

On Wed Apr 15, 2026 at 12:52 PM -03, Shuah Khan wrote:
> On 4/15/26 09:42, Ricardo B. Marli=C3=A8re wrote:
>> On Wed Apr 15, 2026 at 12:40 PM -03, Shuah Khan wrote:
>>> On 4/15/26 07:58, Mark Brown wrote:
>>>> On Fri, Mar 20, 2026 at 03:29:20PM -0300, Ricardo B. Marli=C3=A8re wro=
te:
>>>>> Track failures explicitly in the top-level selftests all/install loop=
s.
>>>>>
>>>>> The current code multiplies `ret` by each sub-make exit status. For
>>>>> example, with `TARGETS=3Dnet`, the implicit `net/lib` dependency runs=
 after
>>>>> `net`, so a failed `net` build can be followed by a successful `net/l=
ib`
>>>>> build and reset the final result to success.
>>>>>
>>>>> Set `ret` to 1 on any non-zero sub-make exit code and keep it sticky,=
 so
>>>>> the top-level make returns failure when any selected selftest target
>>>>> fails.
>>>>
>>>> This patch, which is now in mainline as 7e47389142b8, is breaking a
>>>> bunch of CI systems - at least KernelCI, our Arm internal CI and my
>>>> personal stuff.  It causes the equivalent of FORCE_TARGETS behaviour i=
n
>>>> the top level Makefile, the prior behaviour where the exit status of t=
he
>>>> top level Makefile ignores failures from individual directories is
>>>> desirable since by default we try to build almost all the selftests bu=
t
>>>> between quality issues and build time dependencies it's very common fo=
r
>>>> at least one of them to fail.  With this commit unless the user has
>>>> configured a more restricted set of selftests it would be surprising i=
f
>>>> we manage to get a successful build and install.
>>>>
>>>> As well as being a poor default due to the very high likelyhood of bui=
ld
>>>> failures this also has the undesirable effect of causing a build failu=
re
>>>> in one selftest to cause the whole install target to fail, meaning tha=
t
>>>> the build failure is escallated to a complete lost of coverge for all
>>>> selftests in common CI usage.
>>>>
>>>> This wasn't showing up in my -next build tests since I set FORCE_TARGE=
TS
>>>> and explicitly choose a restricted set of kselftests which actually
>>>> build with my system and configuration.  It was less obvious than it
>>>> should have been with the other systems since they did not expect ther=
e
>>>> to be a complete failure to generate a kselftest tarball and variously
>>>> masked the error or reported it in a manner that looked like an
>>>> infrastructure issue.
>>>
>>> I didn't see it when I did test on linux-next and my repo. I did instal=
l
>>> to catch problems.
>>>
>>> Sorry for not catching this. We can drop this patch.
>>>
>>>>
>>>> It would be really nice to get to the point where we can reasonably do
>>>> this but we're simply not there at the current time.  At the moment if
>>>> people want to see build failures reported at the top level that reall=
y
>>>> needs to be opt in, we have FORCE_TARGETS for that.
>>>
>>> Good point - I will go look and see if we document this in kselftest do=
c
>>> and add it.
>>=20
>> It's not documented. It would have solved my issue, sorry for
>> overlooking this!
>
> Ricardo, Would you like to send me doc patch for this?

Sure, thanks!

>
> thanks,
> -- Shuah