Re: [bpf-next 3/4] selftests/bpf: report failed subtest count in test_progs summary

On 8/3/26 11:31 AM, [email protected] wrote:
>> diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c
>> --- a/tools/testing/selftests/bpf/test_progs.c
>> +++ b/tools/testing/selftests/bpf/test_progs.c
> [ ... ]
>
>> @@ -1726,12 +1731,12 @@
>>   	fclose(env->json);
>>
>>   if (env->not_built_cnt)
>> -	printf("Summary: %d/%d PASSED, %d SKIPPED (%d not built), %d FAILED\n",
>> +	printf("Summary: %d/%d PASSED, %d SKIPPED (%d not built), %d/%d FAILED\n",
>>   	       succ_cnt, sub_succ_cnt, skip_cnt, env->not_built_cnt,
>> -	       fail_cnt);
>> +	       fail_cnt, sub_fail_cnt);
>>   else
>> -	printf("Summary: %d/%d PASSED, %d SKIPPED, %d FAILED\n",
>> -	       succ_cnt, sub_succ_cnt, skip_cnt, fail_cnt);
>> +	printf("Summary: %d/%d PASSED, %d SKIPPED, %d/%d FAILED\n",
>> +	       succ_cnt, sub_succ_cnt, skip_cnt, fail_cnt, sub_fail_cnt);
> A subsystem pattern flags this as potentially concerning: the BPF selftests
> guide asks that test binaries emit TAP-compliant output via kselftest.h
> (ksft_set_plan()/ksft_finished()) rather than hand-formatting their own
> summary lines with printf(), so that kselftest runners and CI can parse the
> results uniformly.
>
> test_progs.c has always had its own reporting framework and does not use
> kselftest.h; this predates the patch by many years and converting it is far
> outside the scope of this change. The patch only widens one existing field on
> an already ad hoc line.
>
> The changelog acknowledges that the field-format change means "anything
> scraping 'N FAILED' out of it needs updating." A search over .github/workflows,
> .github/scripts, ci/, tools/testing/selftests/bpf/*.sh and Documentation/
> turned up no in-tree parser of the test_progs summary line.
>
> Does this output format remain compatible with the consumers you're aware of?

As the Changelog mentions, I don't know of any existing in-tree 
consumers/parser of the existing info.

Thx,
-Vineet

>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/30836252778