Re: Excluding FAILs from UNSUPPORTED test cases

Arsen Arsenović wrote:
> On Monday, 26 September 2022 03:11:43 CEST Jacob Bachmeyer wrote:
>   
>> The tests that it fixed probably need some attention in the form of
>> moving some "preclean" steps to GCC's ${tool}-dg-test.
>>     
> I'll add giving them a look to my TODO.
>   

Please keep me informed about the results of looking into those; if this 
has actually exposed bugs in the GCC testsuite, I may add (as a 
testsuite debugging aid) an option to run ${tool}-dg-prune in an extra 
pass before analyzing the output.  I am tentatively considering a 
variable DG_PRUNE_EARLY_AND_OFTEN that could be set on the runtest 
command line to arrange for this.

>> Keeping the current interface while fixing the {PASS,FAIL}+UNSUPPORTED
>> issue is also possible:  we can build up a list of results from
>> scanning the messages, but not actually report them to the framework
>> until after checking the return value of ${tool}-dg-prune for an
>> "abandon test" flag.  Note that ${tool}-dg-test could produce this
>> flag in lieu of actual output and ${tool}-dg-prune simply pass it
>> along.  (I suspect that the current Tcl list facilities that would
>> enable this were not yet available when dg.exp was written more than
>> 20 years ago.)
>>
>> The main reason I am reluctant to simply accept the "abandon test"
>> flags directly from ${tool}-dg-test is that that would completely
>> break backwards compatibility:  testsuites depending on this would
>> absolutely need DejaGnu 1.6.4, and I think we may be able to get most
>> of the way there (at least getting an UNSUPPORTED result reliably
>> into the summary) with existing code.
>>     
> Right, I didn't think of that.  Indeed, aggregating the results to 
> dispatch them after pruning (but before excess error) should cover all 
> of these cases.
>   

This is still the plan, although I must ask if you know of a reason 
(other than inertia) not to continue aggregating results until the very 
end of the dg-test procedure.

>> Excellent!  It is now tracked as PR58065.  I expect to have a
>> regression test and a preliminary patch in a few days.
>>     
> That's great.  Thank you very much!
>   

My apologies for the delay:  developing unit tests for dg.exp has proven 
to be more involved than initially expected.  (Starting with writing a 
pure-Tcl VFS shim for the internal unit test support infrastructure 
because a getdirs test in utils.test will fail if any more subtrees are 
added to testsuite/runtest.libs.  Why yes, I now plan to eventually also 
make the other tests independent of the actual filesystem now that I 
have had to write this shim.)


-- Jacob