Re: Providing more precise "excess errors" message in DejaGnu

On Fri, 2025-08-29 at 10:40 +0100, Richard Earnshaw via Gcc wrote:
> On 29/08/2025 10:36, Iain Sandoe wrote:
> >=20
> >=20
> > > On 29 Aug 2025, at 10:32, Richard Earnshaw
> > > <[email protected]> wrote:
> > >=20
> > > On 29/08/2025 04:08, Jacob Bachmeyer wrote:
> > > > On 8/28/25 10:10, Jeff Law wrote:
> > > > > On 8/28/25 8:09 AM, Richard Earnshaw (lists) wrote:
> > > > > > On 28/08/2025 15:01, Iain Sandoe wrote:
> > > > > > > [...]
> > > > > >=20
> > > > > > Well really, the compare-tests script should report
> > > > > > duplicate results as a problem as well, since
> > > > > >=20
> > > > > > PASS: abcd
> > > > > > ...
> > > > > > PASS: abcd
> > > > > >=20
> > > > > > is just a dup pass/fail waiting to happen.
> > > > > Yup.=C2=A0 A duplicate testname should be reported.=C2=A0 These c=
ause
> > > > > major headaches if one passes, but the other fails -- it
> > > > > looks like a regression to the comparison scripting we have.
> > > > The problem with detecting duplicate names in the DejaGnu
> > > > framework is that it would add memory overhead that scales with
> > > > the number of tests and DejaGnu tries to avoid that kind of
> > > > unbounded space requirement. (OK, it *is* bounded for any
> > > > finite testsuite, but the idea of a steadily growing memory
> > > > footprint during a test run still bothers me.)
> > > > I suggest that the comparison script GCC uses is probably the
> > > > best place to check for duplicate test names, since that seems
> > > > to also be the script that can be confused by them.
> > >=20
> > > That's exactly what I was suggesting.=C2=A0 Trying to do it in dejagn=
u
> > > would be a nightmare given that we run multiple instances of it
> > > to get parallel testing.
> >=20
> > For the record, I=E2=80=99ve now proposed a BoF for the cauldron on
> > =E2=80=98improving the raw output=E2=80=99 from the testsuite, since th=
at seems at
> > least one place we can make progress - by making the input to the
> > post-processing tools more machine-friendly.=C2=A0 Hopefully the
> > concerns and ideas from this thread can contribute there.
>=20
> I don't know how much maintenance DejaGNU is getting these days, but=20
> certainly something like a --json option to runtest that caused the
> .sum=20
> files to be written as structured data would make some post-
> processing=20
> significantly easier.

I wonder if that json format could be SARIF?  (which might give us some
nice IDE integration and other capabilities "for free")  Perhaps SARIF
is a good fit (or perhaps it isn't).   In any case, it sounds like I
should try and attend the BoF on improving the testsuite output -
thanks for proposing it.

Dave