Re: TAP Directives for XPASS and XFAIL tests with lib/tap-driver.sh

GraphicsMagick uses a custom TAP-test script (patterned on the example 
one from Automake) where there is support for deciding if the test is 
expected to fail based on a list of features.  For example, the support 
for a file format may depend on an optional library and that library is 
a "feature".  The test_command_fn() function has this usage description:

# Test a simple command where pass/fail depends on command exit status
#
# Usage:
#   test_command_fn description [ -F 'feat1 feat2 ...'] command
#
# Where:
#    description -- test description
#    -F          -- optional space-delimited required features
#    command     -- command to execute

A typical test expression looks like:

test_command_fn "MNG ${type}" -F PNG ${MEMCHECK} ${rwfile} -filespec 
"out_${type}_%d" "${SRCDIR}/input_${type}.miff" MNG

so in the above one can see that "PNG" is a feature (in this case 
depending on libpng being present).

This strategy has been working well for a great many years.

The ability to supply a custom test functions script which is matched to 
the software being tested is quite useful.

The idea of adding a directive that certain tests are expected to fail 
is interesting, but the immediate feature-based approach seems much 
easier (and more efficient) than trying to add indirect support, such as 
via an algorithm or database lookup.

Bob

On 6/30/25 20:23, Jacob Bachmeyer wrote:
> On 6/30/25 16:41, Soham wrote:
>> Hello,
>>
>> I recently was using the tap driver provided by automake, and noticed 
>> that there currently is no way to declare a test as xfail or xpass 
>> apart from marking it as "todo". The test ouput however does produce 
>> rows for XPASS and XFAIL, and the existing TRS format already 
>> supports both.
>>
>> Since the TAP specification allows for custom directives [1], I was 
>> wondering if there was interest in a patch adding these to 
>> `lib/tap-driver.sh`. I would add "XPASS" and "XFAIL" directives, and 
>> if the test did not pass/fail as expected, it would result in an ERROR.
>
> Perhaps Autotest uses ERROR differently, but the better way to think 
> about it is expected-vs.-unexpected results.  A test marked XFAIL that 
> fails is expected to fail, but a test marked XFAIL that passes is a 
> surprise XPASS.
>
> A better solution is to add a directive that declares that certain 
> tests are expected to fail.  The TAP driver then matches the actual 
> results with the expected results.
>
>
> -- Jacob
>
>
>