Re: Problem with spuriously failing gcc tests with dg-output.

On 02.06.2017 11:42, Christophe Lyon wrote:
> On 2 June 2017 at 11:35, Georg-Johann Lay <[email protected]> wrote:
>> Hi, when I am running gcc test suite
>>
>> $ make check-gcc RUNTESTFLAGS='-all ubsan.exp=float-cast-overflow-2.c'
>>
>>                 === gcc tests ===
>>
>> Schedule of variations:
>>     unix
>>
>> Running target unix
>> Using /usr/share/dejagnu/baseboards/unix.exp as board description file for
>> target.
>> Using /usr/share/dejagnu/config/unix.exp as generic interface file for
>> target.
>> Using /home/georg/gnu/gcc.gnu.org/trunk/gcc/testsuite/config/default.exp as
>> tool-and-target-specific interface file.
>> Running
>> /home/georg/gnu/gcc.gnu.org/trunk/gcc/testsuite/gcc.dg/ubsan/ubsan.exp ...
>> PASS: c-c++-common/ubsan/float-cast-overflow-2.c   -O0  (test for excess
>> errors)
>> PASS: c-c++-common/ubsan/float-cast-overflow-2.c   -O0  execution test
>> FAIL: c-c++-common/ubsan/float-cast-overflow-2.c   -O0  output pattern test
>> PASS: c-c++-common/ubsan/float-cast-overflow-2.c   -O1  (test for
>> ...
>>
>> I am getting spurious FAILs.  Rerunning the same tests might come up with
>> more or less FAILs / PASSes.
>>
>> The tests have "lots" of dg-output directives
>>
>> $ cat gcc/testsuite/c-c++-common/ubsan/float-cast-overflow-2.c | tail -n 3
>> /* { dg-output "\[^\n\r]*runtime error: value -?nan is outside the range of
>> representable values of type '__int128 unsigned'\[^\n\r]*(\n|\r\n|\r)" } */
>> /* { dg-output "\[^\n\r]*runtime error: value inf is outside the range of
>> representable values of type '__int128 unsigned'\[^\n\r]*(\n|\r\n|\r)" } */
>> /* { dg-output "\[^\n\r]*runtime error: value -inf is outside the range of
>> representable values of type '__int128 unsigned'" } */
>>
>> https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/testsuite/c-c%2B%2B-common/ubsan/float-cast-overflow-2.c?revision=212755&view=markup#l1
>>
>> In the according log file, some of the "PASS..." output is messed up with
>> the output from dg-output, making the test fail:
>>
>> $ cat testsuite/gcc/gcc.log
>> PASS: c-c++-common/ubsan/float-cast-overflow-2.c   -O0  (test for excess
>> errors)
>> Setting LD_LIBRARY_PATH to : [cut]
>> spawn [open ...]
>>
>> /home/georg/gnu/gcc.gnu.org/trunk/gcc/testsuite/c-c++-common/ubsan/float-cast-overflow-2.c:20:3:
>> runtime error: value 1.70141e+38 is outside the range of representable
>> values of type '__int128'
>> /home/georg/gnu/gcc.gnu.org/trunk/gcc/testsuite/c-c++-common/ubsan/float-cast-overflow-2.c:20:3:
>> runtime error: value 1.70141e+38 is outside the range of representable
>> values of type '__int128'
>> [...]
>> /home/georg/gnu/gcc.gnu.org/trunk/gcc/testsuite/c-c++-common/ubsan/float-cast-overflow-2.c:25:3:
>> runtime error: value 3.40282e+38 is outside the range of representable
>> values of type '__int128 unsigned'
>> /home/georg/gnu/gcc.gnu.org/trunk/gcc/testsuite/c-c++-common/ubsan/float-cast-overflow-2.c:25:3:
>> runtime error: value 3.40282e+38 is outside the range of representable
>> values of type '__int128 unsigned'
>> /home/georg/gnu/gcc.gnu.org/trunk/gcc/testsuite/c-c++-common/ubsan/float-cast-overflow-2.c:25:3:
>> runtime error: PASS: c-c++-common/ubsan/float-cast-overflow-2.c   -O0
>> execution test
>>
>> BUMMER! The PASS output comes at the wrong place and messes up the dg-output
>>
>> FAIL: c-c++-common/ubsan/float-cast-overflow-2.c   -O0  output pattern test
>> Output was:
>> c-c++-common/ubsan/float-cast-overflow-2.c:20:3: runtime error: value
>> 1.70141e+38 is outside the range of representable values of type '__int128'
>> c-c++-common/ubsan/float-cast-overflow-2.c:20:3: runtime error: value
>> 1.70141e+38 is outside the range of representable values of type '__int128'
>> [...]
>> c-c++-common/ubsan/float-cast-overflow-2.c:25:3: runtime error: value
>> 3.40282e+38 is outside the range of representable values of type '__int128
>> unsigned'
>> c-c++-common/ubsan/float-cast-overflow-2.c:25:3: runtime error: value
>> 3.40282e+38 is outside the range of representable values of type '__int128
>> unsigned'
>> c-c++-common/ubsan/float-cast-overflow-2.c:25:3: runtime error:
>> Should match:
>>
>> Output is missing after "runtime error: " which is the place where the
>> "PASS" message was emit.
>>
>> Any ideas what can cause such fails?  It's on a 8-core x86_64 machine, but
>> the testsuite is not started in parallel mode.
>>
>> $ runtest -version
>> Expect version is       5.45
>> Tcl version is          8.6
>> Framework version is    1.5.3
>>
>> Does this ring a bell somewhere?  What could shred the output?
>>
>
> Maybe it's similar to what we discussed in:
> https://lists.gnu.org/archive/html/dejagnu/2016-03/msg00034.html
>
> Christophe

Wow, that was fast!

What's strange is that GCC folks appears not to have such problems, it's 
not uncommon to run gcc tests on x86_64 host.  The only pointer I found 
was a referrer to a Tcl bug in Tcl 8.6 and fixed in 8.6.1

https://gcc.gnu.org/install/prerequisites.html

$ echo 'puts [info patchlevel];exit 0' | tclsh
8.6.5

I tried to downgrade Tcl to 8.5.19, but runtest -version still reports 
"8.6".

Johann