Re: Excluding FAILs from UNSUPPORTED test cases

Arsen Arsenović wrote:
> Hi,
>
> On Saturday, 24 September 2022 06:24:56 CEST you wrote:
>   
>>> But when it's unsupported because ${tool}-dg-prune returns
>>> "::unsupported::" the individual checks within the file still appear
>>> in the results. I realise that in the target selector case, the test
>>> is just skipped before doing anything, and in the prune case we
>>> don't
>>> know that it's unsupported until *after* testing it. But it seems to
>>> me that ideally the individual checks would get "retroactively
>>> skipped" if tool-dg-prune returns one of ::untested::,
>>> ::unresolved::, or ::unsupported::. Maybe that's not easy to do
>>> though.
>>>       
>> This actually looks like a bug in dg.exp to me.  The output from
>> ${tool}-dg-test is first collected, all at once, (line 696) and then
>> analyzed (lines 701-749).  I would expect ${tool}-dg-prune to have an
>> opportunity to modify the text to be analyzed /before/ the analysis is
>> done, but the current code checks for messages first, /then/ calls
>> ${tool}-dg-prune (and the framework's prune_warnings procedure) to
>> remove irrelevant output, then declares excess errors if the final
>> text is nonempty.
>>     
> This is how I understood the output analysis routine: remove expected 
> lines first (i.e. those matched by dg-messages), then lines that 
> shouldn't be a FAIL but are noise or such instead (i.e. prune), which 
> allows ${tool}-dg-prune and dg-prune-output to be less careful about 
> what they remove (as they don't have to worry about removing diagnostics 
> that should be matched by dg-{error,warning}).
> Changing ${tool}-dg-prune to execute earlier might be a bit too drastic, 
> because it could expose some overzealous pruning routines.

Running ${tool}-dg-prune earlier works both ways, allowing the callback 
prune routine to also remove or modify any output that might confuse the 
message-matching mechanism.  Really, given the documented (such as it 
is) feature for ${tool}-dg-prune to be able to declare the entire test 
unsupported, I would expect it to filter the results before messages are 
checked.

Overzealous pruning routines are bugs in the testsuites that have them, 
since they would be very likely to hide /actual/ excess errors, thereby 
causing the test for excess errors to PASS when it should FAIL.

>  On the other 
> hand, giving ${tool}-dg-test a way to terminate analysis early, before 
> all the output processing, could maybe constrained enough to not change 
> what existing code does (which is why I suggested it earlier).
> What do you think?
>   
Tcl provides a way to do that (I think) if you really want:

    unsupported "freestanding libstdc++"
    return -code return

...but this would be a bit of a hack.


-- Jacob