Re: TAP Directives for XPASS and XFAIL tests with lib/tap-driver.sh

On 7/5/25 14:36, Soham wrote:
> Thanks for the replies everyone!
>
> On 7/3/25 21:40, Jacob Bachmeyer wrote:
>> [...]
>> TAP *should* *not* have semantics for that.  TAP scripts are supposed 
>> to be runnable directly or through trivial harnesses like piping the 
>> output through `grep 'not ok'`.  If the meaning of "ok" and "not ok" 
>> could be inverted on a test-by-test basis, there would be room for 
>> endless confusion.
>
> Yeah this makes more sense, thanks for the insight. It does make sense 
> that
> changing the meaning of ok/not ok is not a good idea, though the 
> binary choice
> does feel limiting.

The binary choice is *intended* to be limiting.  Either the behavior 
meets the specification ("ok") or it does not meet the specification 
("not ok").

The only other options are failures in the testsuite itself (DejaGnu 
"UNRESOLVED"; TAP "Bail out!"), an incomplete test (DejaGnu "UNTESTED"; 
TAP "TODO"), or lack of local support rendering the test meaningless 
(DejaGnu "UNSUPPORTED"; TAP "SKIP").

> A renewed proposal could be adding an " # XFAIL " directive, that simply
> moves the counting of the test to the XFAIL: row of the output of 
> `make check`.
> That would accomplish both the ability to programtically deem tests as 
> "XFAIL"
> and would preserve the semantics of ok/not ok. Please let me know 
> would be
> worthwhile.

As long as the understanding is that an XFAIL test is a test that 
*should* pass but currently fails due to bugs here or elsewhere.


-- Jacob