Re: [PATCH v2 6/6] Makerules: add 'make check-parallel' to run tests without serialization

On 03/07/26 08:09, Sam James wrote:
> Adhemerval Zanella <[email protected]> writes:
> 
>> The default 'make check' serializes the timing-sensitive test runs: the
>> threading (nptl, or htl on Hurd) and realtime (rt) subdirectories run
>> with .NOTPARALLEL and are ordered after the rest of the test run, so they
>> are not perturbed by competing machine load.
>>
>> Add a check-parallel (and xcheck-parallel) variant for when that is not
>> wanted -- an idle machine, or a run where the extra throughput is worth
>> the risk of flakiness in the timing-sensitive tests.
>>
> 
> Perhaps link to
> https://inbox.sourceware.org/libc-alpha/[email protected]/

Ack.

> for context in the commit message. I think ultimately we want to try do
> something better for these, like adding some classifier for them as CPU
> intensive or timing sensitive.

Yeah, it is my backlog to check a way classify such tests with a special
rule (like tests-timing or something like that) and run each of them
serially.

> 
> It also reminds me a bit of some of the xcheck discussion:
> https://inbox.sourceware.org/libc-alpha/5816153.DvuYhMxLoT@pinacolada/.

For xcheck I think some tests could be rewritten to avoid being 
timing-sensitive, but some does require some kernel support that does
require running with extra capabilities (like CAP_SYS_NICE for sccheduling
priority). 

> 
>> A new serialize-tests flag (default yes, defined in Makeconfig) gates both
>> the per-subdirectory .NOTPARALLEL and the top-level run-time ordering;
>> check-parallel just runs the suite with serialize-tests=no, so every test
>> program builds and runs at full concurrency in a single pass.
>>
>> 'make check' and its default behavior are unchanged.
> 
> Can you make sure you document this on the wiki as well at
> https://sourceware.org/glibc/wiki/Testing/Testsuite once in?

I will do it.

> 
> Reviewed-by: Sam James <[email protected]>


Thanks.

> 
>> ---
>>  Makeconfig    |  8 ++++++++
>>  Makefile      |  3 +++
>>  Makerules     | 13 +++++++++++++
>>  htl/Makefile  |  5 +++--
>>  nptl/Makefile |  5 +++--
>>  rt/Makefile   |  5 +++--
>>  6 files changed, 33 insertions(+), 6 deletions(-)
>>
>> diff --git a/Makeconfig b/Makeconfig
>> index 8fe7217dd7c..6cd590cf216 100644
>> --- a/Makeconfig
>> +++ b/Makeconfig
>> @@ -782,6 +782,14 @@ run-built-tests = yes
>>  endif
>>  endif
>>  
>> +# Whether the timing-sensitive test runs are serialized: each is run with
>> +# .NOTPARALLEL and, at the top level, ordered after the rest of the test run.
>> +# This is the default; 'make check-parallel' clears it to run every test
>> +# concurrently.
>> +ifndef serialize-tests
>> +serialize-tests = yes
>> +endif
>> +
>>  # Whether to build the static math tests
>>  ifndef build-math-static-tests
>>  build-math-static-tests = no
>> diff --git a/Makefile b/Makefile
>> index 5f2293dfd7d..a38f11facf6 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -594,7 +594,9 @@ $(objpfx)testroot.pristine/install.stamp: | others
>>  # 'make subdir/tests' is left alone.  And it only orders the test run
>>  # (run-built-tests=yes); the "build the tests" pass (run-built-tests=no)
>>  # is left fully parallel, so every test program still builds concurrently.
>> +# serialize-tests=no ('make check-parallel') drops the ordering entirely.
>>  ifeq ($(run-built-tests),yes)
>> +ifeq (yes,$(serialize-tests))
>>  ifneq (,$(filter tests xtests check xcheck,$(MAKECMDGOALS)))
>>  +late-test-subdirs := $(filter nptl htl,$(subdirs)) $(filter rt,$(subdirs))
>>  +test-run-prev := \
>> @@ -604,6 +606,7 @@ $(foreach d,$(+late-test-subdirs),\
>>    $(eval +test-run-prev += $(d)/tests))
>>  endif
>>  endif
>> +endif
>>  
>>  ifeq (yes,$(build-shared))
>>  # The top-level libc.so and linkobj/libc_pic.a rules list these
>> diff --git a/Makerules b/Makerules
>> index 318a916eef0..d7ab610b294 100644
>> --- a/Makerules
>> +++ b/Makerules
>> @@ -1214,6 +1214,19 @@ check: tests
>>  xcheck: xtests
>>  endif
>>  
>> +# 'make check-parallel' runs the whole suite with maximum concurrency:
>> +# serialize-tests=no drops the per-subdirectory .NOTPARALLEL and the run-time
>> +# ordering, so every test builds and runs in parallel.  A single pass suffices
>> +# (there is no .NOTPARALLEL to work around, so the test programs already build
>> +# concurrently).
>> +# This is faster on an idle machine, at the cost of possible flakiness in the
>> +# timing-sensitive tests under the heavier load.
>> +.PHONY: check-parallel xcheck-parallel
>> +check-parallel:
>> +	$(MAKE) serialize-tests=no tests
>> +xcheck-parallel:
>> +	$(MAKE) serialize-tests=no xtests
>> +
>>  # Also handle test inputs in sysdeps.
>>  vpath %.input $(sysdirs)
>>  
>> diff --git a/htl/Makefile b/htl/Makefile
>> index 0e5a8b8f689..1035b075649 100644
>> --- a/htl/Makefile
>> +++ b/htl/Makefile
>> @@ -258,8 +258,9 @@ else
>>  $(addprefix $(objpfx),$(tests) $(test-srcs)): $(srcdir)/libpthread_syms.a $(objpfx)libpthread.a
>>  endif
>>  
>> -# The tests here better do not run in parallel.
>> -ifeq ($(run-built-tests),yes)
>> +# The tests here better do not run in parallel, unless serialize-tests is
>> +# cleared (make check-parallel).
>> +ifeq (yes-yes,$(run-built-tests)-$(serialize-tests))
>>  ifneq ($(filter %tests,$(MAKECMDGOALS)),)
>>  .NOTPARALLEL:
>>  endif
>> diff --git a/nptl/Makefile b/nptl/Makefile
>> index 02862d1c04b..97a0fb7f658 100644
>> --- a/nptl/Makefile
>> +++ b/nptl/Makefile
>> @@ -738,8 +738,9 @@ tst-audit-threads-ENV = LD_AUDIT=$(objpfx)tst-audit-threads-mod1.so
>>  tst-setuid1-static-ENV = \
>>    LD_LIBRARY_PATH=$(ld-library-path):$(common-objpfx)elf:$(common-objpfx)nss
>>  
>> -# The tests here better do not run in parallel.
>> -ifeq ($(run-built-tests),yes)
>> +# The tests here better do not run in parallel, unless serialize-tests is
>> +# cleared (make check-parallel).
>> +ifeq (yes-yes,$(run-built-tests)-$(serialize-tests))
>>  ifneq ($(filter %tests,$(MAKECMDGOALS)),)
>>  .NOTPARALLEL:
>>  endif
>> diff --git a/rt/Makefile b/rt/Makefile
>> index 24e43a87599..f39e9674e58 100644
>> --- a/rt/Makefile
>> +++ b/rt/Makefile
>> @@ -111,8 +111,9 @@ $(objpfx)librt.so: $(shared-thread-library)
>>  tst-mqueue7-ARGS = -- $(host-test-program-cmd)
>>  
>>  # The timer and message-queue tests here are timing-sensitive and better
>> -# do not run in parallel.
>> -ifeq ($(run-built-tests),yes)
>> +# do not run in parallel, unless serialize-tests is cleared (make
>> +# check-parallel).
>> +ifeq (yes-yes,$(run-built-tests)-$(serialize-tests))
>>  ifneq ($(filter %tests,$(MAKECMDGOALS)),)
>>  .NOTPARALLEL:
>>  endif