Re: TAP Directives for XPASS and XFAIL tests with lib/tap-driver.sh

Thanks for the replies everyone!

On 7/3/25 21:40, Jacob Bachmeyer wrote:
> On 7/3/25 18:11, Russ Allbery wrote:
>> Karl Berry <[email protected]> writes:
>>
>> [...]
> 
> This confusion comes from not clearly thinking through what the test 
> *is*.  A proper test verifies conformance to a specification.  The 
> example from the DejaGnu manual is a specification "The sun shall 
> shine." and a test "The sun is shining." which passes or fails depending 
> on whether it is day or night.
> 

Well, my original thinking was writing tests with invalid input for a given
function, and have it expectedly "not ok". This makes some sense because
one would expect a function given bogus (or perhaps edge case) arguments
to fail. The discrepancy is whether to report a failure as "ok", or to 
mark it
as "not ok" with a directive for the harness.

> In this case, the specification is "The syscall shall return an error if 
> given an invalid argument." and the test provides an invalid argument 
> and expects an error return code.  The test *passes* ("ok" in TAP) *if* 
> *and* *only* *if* the syscall reports failure when given an invalid 
> argument.

>[...]
> TAP *should* *not* have semantics for that.  TAP scripts are supposed to 
> be runnable directly or through trivial harnesses like piping the output 
> through `grep 'not ok'`.  If the meaning of "ok" and "not ok" could be 
> inverted on a test-by-test basis, there would be room for endless 
> confusion.
> 

Yeah this makes more sense, thanks for the insight. It does make sense that
changing the meaning of ok/not ok is not a good idea, though the binary 
choice
does feel limiting.

A renewed proposal could be adding an " # XFAIL " directive, that simply
moves the counting of the test to the XFAIL: row of the output of `make 
check`.
That would accomplish both the ability to programtically deem tests as 
"XFAIL"
and would preserve the semantics of ok/not ok. Please let me know would be
worthwhile.

Infact my original motivation is/was to see if I could make use of the 
various
rows put out by `make check`, which admittedly is rather cosmetic.