bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case

Rob Savoye wrote:
> On 6/25/20 7:53 PM, Jacob Bachmeyer wrote:
>   
>> The original rationale for using UNRESOLVED here was that DejaGnu
>> converts a test result to UNRESOLVED if too many errors or warnings were
>> produced.  The DejaGnu manual indicates (section "A POSIX Compliant Test
>> Framework") that UNRESOLVED is correct for a test where execution was
>> interrupted or was set up incorrectly.  UNTESTED is specifically listed
>> as a placeholder for an as-yet-unwritten testcase.  A "typical" GDB
>> testsuite run has almost a hundred UNTESTED results but (without this
>> patch) zero UNRESOLVED results.
>>
>> To me, this seems that UNRESOLVED is correct here, or that the manual
>> has an error.
>>     
>
>   So having written both, there is always the chance I've interpreted
> things differently. If the test is interrupted, or has errors/warnings,
> like timing problems for example, then it's UNRESOLVED. But a bug in Tcl
> code enough to trigger unknown is UNTESTED, as the test never really
> ran. I'd go with which definition the toolchain teams prefer, as this
> effects validation testruns. I could go either way.
>   

I see UNTESTED as a placeholder for a yet-to-be-written test; DejaGnu 
does not produce an error exit status after reporting UNTESTED.

Once Tcl has begun to execute "source $test_file_name", I see the test 
as running from the framework's perspective, so an error exit should be 
UNRESOLVED:  we do not know the result that would have been produced had 
the Tcl error not occurred.

>   I don't know if POSIX 1003.3 is still considered a standard, that was
> a long time ago. The testsuites were primarily focused on XOpen and
> POSIX conformance tests, not toolchains. I was
> on the standards committee, so used them as a model cause they seemed a
> good standard. But the details of the interpretation can now be whatever
> we want it to be. We're probably the only ones left running TET
> compliant testsuites anyway. :-)
>   

POSIX 1003.3 appears to have been withdrawn.

>   Anyway, whether a test run aborts on an error or not is not defined,
> so it's up to us to decide.

Apparently, GDB has baked an assumption that the test run will not 
abort, ever, into their testsuite, and we had never documented 
circumstances that cause DejaGnu to abort, so I cannot really answer 
that with "your assumption is wrong".


-- Jacob



_______________________________________________
Bug-dejagnu mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/bug-dejagnu