Re: Providing more precise "excess errors" message in DejaGnu

> On 28 Aug 2025, at 16:10, Jeff Law via Gcc <[email protected]> wrote:
>=20
>=20
>=20
> On 8/28/25 8:09 AM, Richard Earnshaw (lists) wrote:
>> On 28/08/2025 15:01, Iain Sandoe wrote:
>>>=20
>>>=20
>>>> On 28 Aug 2025, at 14:36, Jeff Law via Gcc <[email protected]> wrote:
>>>>=20
>>>>=20
>>>>=20
>>>> On 8/28/25 5:42 AM, Richard Biener via Gcc wrote:
>>>>> On Thu, Aug 28, 2025 at 1:12=E2=80=AFPM Rainer Orth via Gcc =
<[email protected]> wrote:
>>>>>>=20
>>>>>> Hi Sam,
>>>>>>=20
>>>>>>> When a test fails with 'excess errors', there's often only one =
actual
>>>>>>> error (an excess "(error|warning|note):") and it'd be nice to =
not have
>>>>>>> to dig in the .log files to fish that out.
>>>>>>=20
>>>>>> I think such a move would be a bad mistake.  Consider ICEs where =
you
>>>>>> have something like
>>>>>>=20
>>>>>> FAIL: gcc.c-torture/compile/pr35318.c   -O0  (test for excess =
errors)
>>>>>> Excess errors:
>>>>>> =
/vol/gcc/src/hg/master/local/gcc/testsuite/gcc.c-torture/compile/pr35318.c=
:9:1: error: unrecognizable insn:
>>>>>> (insn 13 25 26 2 (parallel [
>>>>>>             (set (reg:DF 10 %o2 [orig:112 x ] [112])
>>>>>>                 (asm_operands/v:DF ("") ("=3Dr,r") 0 [
>>>>>>                         (reg:SI 11 %o3 [orig:112 x+4 ] [112])
>>>>>>                         (mem/c:DF (plus:SI (reg/f:SI 30 %fp)
>>>>>>                                 (const_int -24 =
[0xffffffffffffffe8])) [3 %sfp+-24 S8 A64])
>>>>>> [and many more lines...]
>>>>>>=20
>>>>>> This would clutter the output beyond recognition, especially if =
this is
>>>>>> a torture test which is run at several optimization options.
>>>>>>=20
>>>>>> Excess errors, like all others, always require further =
investigation.
>>>>>> In my experience, digging the full error messages from the .log =
files is
>>>>>> usually the smallest part of that.  You often even have to rerun =
the
>>>>>> compilation manually to also get the parts that are filtered out =
by the
>>>>>> prune procs.
>>>>> I find the classification this would provide useful, like I like =
the
>>>>> (internal compiler error) classification we already have.  I would
>>>>> of course not duplicate all of th eabove message but only
>>>>> '(unrecognizable insn)' in the above case.
>>>> It's not the only consideration, but keep in mind that such output =
is not stable and will cause some headaches with scripting that compares =
two summary files.
>>>=20
>>> This is something that I think we want to tackle anyway ..
>>>>=20
>>>> Even with the bit of instability to the line number in the ICE =
message, I do find the ICE classification useful as well.
>>>=20
>>> =E2=80=A6 the classification is useful, but the false positive =
=E2=80=9Cnew fail / old fail went away=E2=80=9D pairs are a real =
nuiscance .. hopefully we can have some brainstorming @cauldron about =
ways to deal with this (e.g. fuzzy matching or some smart way to discard =
the twinkling line numbers).
>>>=20
>> Well really, the compare-tests script should report duplicate results =
as a problem as well, since
>> PASS: abcd
>> ...
>> PASS: abcd
>> is just a dup pass/fail waiting to happen.
> Yup.  A duplicate testname should be reported.  These cause major =
headaches if one passes, but the other fails -- it looks like a =
regression to the comparison scripting we have.
>=20
> Getting to the point where every test has a unique name would be good =
on many levels.  I can't help but think back to QMTest which tried to =
solve the enumeration problem along with others.  I wasn't too =
supportive at the time, but in retrospect, that was probably a mistake.

FWIW I was (and still am) about to volunteer to start a BoF on improving =
the testsuite output (with the infra we currently have) - there are some =
concrete ideas - but it probably needs some brainstorming too.   This is =
distinct from any BoF on improving test coverage or =E2=80=9CCI=E2=80=9D =
in general  =E2=80=A6 seems like there should be some interest in such a =
discussion.

Iain

>=20
> jeff