Re: Excluding FAILs from UNSUPPORTED test cases

Arsen Arsenović wrote:
> On Friday, 30 September 2022 04:02:25 CEST Jacob Bachmeyer wrote:
>   
>> Please keep me informed about the results of looking into those; if
>> this has actually exposed bugs in the GCC testsuite, I may add (as a
>> testsuite debugging aid) an option to run ${tool}-dg-prune in an
>> extra pass before analyzing the output.  I am tentatively considering
>> a variable DG_PRUNE_EARLY_AND_OFTEN that could be set on the runtest
>> command line to arrange for this.
>>     
> I think this is actually a false positive when comparing, as it was an 
> XFAIL turned into an XPASS (but, again, it was attempting to detect a 
> note: that got pruned because pruning happened earlier).  There's five 
> relevant files, if you'd like to look yourself, in the gcc/testsuite 
> directory inside the GCC tree:
> - c-c++-common/pr69543-3.c
> - g++.dg/other/error8.C
> - g++.dg/template/dtor7.C
> - g++.old-deja/g++.pt/overload7.C
> - gcc.dg/uninit-pr89230-1.c
>   

This suggests that early pruning would actually hide bugs in GCC:  if 
the compiler is not supposed to produce the message in question, then 
pruning it causes the test to spuriously pass and the XFAIL result is 
correct.  (I have not yet developed the level of skill to do much more 
on GCC than building and using it, so I am afraid that those tests are 
not particularly meaningful to me.  :-/  )

Or did early pruning expose a bug in the GCC testsuite:  the pruning 
function is overzealous and removes messages that GCC should not have 
produced and therefore should have been left for DejaGnu to detect?

> [...]
>   
>> My apologies for the delay:  developing unit tests for dg.exp has
>> proven to be more involved than initially expected.  (Starting with
>> writing a pure-Tcl VFS shim for the internal unit test support
>> infrastructure because a getdirs test in utils.test will fail if any
>> more subtrees are added to testsuite/runtest.libs.  Why yes, I now
>> plan to eventually also make the other tests independent of the
>> actual filesystem now that I have had to write this shim.)
>>     
> No worries!  That shim sounds quite neat.
>   

If you want to see it, the shim (as it stands thus far) is in the 
DejaGnu repository at Savannah on the PR58065 branch; as of this 
writing, I have not yet committed the dg.test file that uses it.


-- Jacob