Re: Providing more precise "excess errors" message in DejaGnu

On 2025-08-28 16:18, Thiago Jung Bauermann via Gcc wrote:
> "Richard Earnshaw (lists)" <[email protected]> writes:
> 
>> On 28/08/2025 15:01, Iain Sandoe wrote:
>>>
>>> … the classification is useful, but the false positive “new fail / old fail went away”
>>> pairs are a real nuiscance .. hopefully we can have some brainstorming @cauldron about
>>> ways to deal with this (e.g. fuzzy matching or some smart way to discard the twinkling
>>> line numbers).
>>>
>>
>> Well really, the compare-tests script should report duplicate results as a problem as
>> well, since
>>
>> PASS: abcd
>> ...
>> PASS: abcd
>>
>> is just a dup pass/fail waiting to happen.
> 
> GDB has some magic to detect and report duplicate test names during the
> DejaGNU run:
> 
> https://sourceware.org/git?p=binutils-gdb.git;a=blob;f=gdb/testsuite/lib/check-test-names.exp;h=049addd49a93744a832ec0fb5249cd59f3db74f1;hb=refs/heads/master#l64
> 

It might be worth to show how it looks in practice too.  

And I'll show other neat features the GDB testsuite has, in case they pique your interest.

Here's an example of the duplicates detection:

$ cat testsuite/gdb.base/dup.exp 
clean_restart

gdb_test "print 1"
gdb_test "print 1"

$ make check TESTS="gdb.base/dup.exp"

Running .../src/gdb/testsuite/gdb.base/dup.exp ...
DUPLICATE: gdb.base/dup.exp: print 1             <<< NEW (goes on screen in addition to gdb.sum for extra visibility)

                === gdb Summary ===

# of expected passes            2
# of duplicate test names       1                <<< NEW

$ cat testsuite/gdb.sum 
...
Running .../src/gdb/testsuite/gdb.base/dup.exp ...
PASS: gdb.base/dup.exp: print 1
PASS: gdb.base/dup.exp: print 1
DUPLICATE: gdb.base/dup.exp: print 1
...
$

GDB used to have hundreds of duplicated test names, and we've since fixed them all, and it is now
hard to introduce any new instance because that mechanism above flags it immediately.

GDB's testsuite also has similar machinery to flag if a source or build path appears in gdb.sum,
as those makes it more difficult to compare results across builds.  E.g.:

     $ make check TESTS="gdb.base/sometest.exp"
     ...
      PASS: gdb.base/sometest.exp: Loaded /path/to/build/testsuite/foo.exe
      PATH: gdb.base/sometest.exp: Loaded /path/to/build/testsuite/foo.exe

                     === gdb Summary ===
    
     # of paths in test names       1       <<< NEW


It also detects if we end up with stray core dumps after a test run (meaning, GDB crashed), the
latter useful because sometimes gdb crashes _after_ dejagnu closes gdb.  e.g.:

     $ make check TESTS="gdb.server/connect-with-no-symbol-file.exp"
     ...
                     === gdb Summary ===
    
     # of unexpected core files      2       <<<< NEW
     # of expected passes            8


All of this works with parallel testing as well, where we invoke multiple expect processes in parallel,
and have a mechanism that lets you mark some tcl procedures as special such that if any of the expect
processes calls such a proc, the result is cached and reused by the other processes that later call the
same procedure with the same arguments.

Another thing the GDB testsuite does is treat any info in trailing parenthesis as "extra" info.  For
example, these three are all considered the same test:

  PASS: foo
  FAIL: foo (timeout)
  FAIL: foo (internal-error)

Pedro Alves