Re: TAP Directives for XPASS and XFAIL tests with lib/tap-driver.sh

Hello, Thanks for the input!

On 7/1/25 15:24, Bob Friesenhahn wrote:
> GraphicsMagick uses a custom TAP-test script (patterned on the example 
> one from Automake) where there is support for deciding if the test is 
> expected to fail based on a list of features.  For example, the support 
> for a file format may depend on an optional library and that library is 
> a "feature".  The test_command_fn() function has this usage description:
> 
> # Test a simple command where pass/fail depends on command exit status
> #
> # Usage:
> #   test_command_fn description [ -F 'feat1 feat2 ...'] command
> #
> # Where:
> #    description -- test description
> #    -F          -- optional space-delimited required features
> #    command     -- command to execute

Well, I forgot to clarify in the original email, but my particular target
use-case is is "unit tests" written in code, say for a particular 
function or
class. In that case it gets cumbersome to have one binary per test 
(atleast IMO).
My idea is to add support for a TAP output like so:

[not] ok 5 - Call sys_open with negative file descriptor # XFAIL


Where the trailing XFAIL directive would put it in the output for XFAIL,
and consider the test "passing". If it were to be an "ok" it would count 
as a
failing test.


> A typical test expression looks like:
> 
> test_command_fn "MNG ${type}" -F PNG ${MEMCHECK} ${rwfile} -filespec 
> "out_${type}_%d" "${SRCDIR}/input_${type}.miff" MNG
> 
> so in the above one can see that "PNG" is a feature (in this case 
> depending on libpng being present).
> 
> This strategy has been working well for a great many years.
> 
> The ability to supply a custom test functions script which is matched to 
> the software being tested is quite useful.
> 
> The idea of adding a directive that certain tests are expected to fail 
> is interesting, but the immediate feature-based approach seems much 
> easier (and more efficient) than trying to add indirect support, such as 
> via an algorithm or database lookup.

Well the algorithm I had in mind is quite simple - if a test out starts with
"ok NUM ... " but contains directive "# XFAIL" then its a failing test, 
otherwise
is a "XFAIL" test which is considered passing.

Essentially its encoding the expected-fail information in the TAP 
message itself
rather than in the surrounding runner script or harness.