Test::Buildder v. Test2 subtests

--PEIAKu/WMn1b1Hv9
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I've hit a nasty (to me) difference between Test2::API::run_subtest and
Test::Builder::subtest.  Shout out to Matthew Horsfall for helping localize=
 the
problem.  Here's my trivial reproducer:

  #!perl
  use strict;
  use warnings;
  use Test::More;
  use Test2::API;

  my $code =3D sub {
    if (fork) {
      wait;
      cmp_ok($?, '!=3D', 0, "subprocess died");
    } else {
      die "123";
    };
  };

  my $call =3D $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest;
  $call->(test =3D> $code);
  done_testing;

When running with Test::More...

  ~$ perl foo 1
  # Subtest: test
  123 at foo line 12.
      ok 1 - subprocess died
      1..1
  ok 1 - test
  1..1

The die in the subprocess causes the child process to exit nonzero and the
tests run normally.  Great!  This is how my tests work.  When I switch them
to Test2::API::run_subtest, though, I get this:

  ~$ perl foo 0
  # test
  not ok 1 - test
  # Failed test 'test'
  # at foo line 17.
  # Caught exception in subtest: 123 at foo line 12.
  1..1
      not ok 1 - subprocess died
      #   Failed test 'subprocess died'
      #   at foo line 10.
      #          got: 0
      #     expected: anything else
      1..1
  not ok 1 - test
  # Failed test 'test'
  # at foo line 17.
  1..1
  # Looks like you failed 1 test of 1.

In the forked code, the die causes the subtest to end in failure *in the fo=
rk*
meaning we get a repeat result for test 1: one from the child, one from the
parent.  Also, the child is now exiting zero instead of nonzero, despite the
fact that it threw an exception.

It gets worse [for me]!  Above, I had a cmp_ok on $? that failed, so I had
duplicate *not ok* results for test 1.  If I replace that cmp_ok with a pas=
s, I
get duplicate *conflicting* results for test 1... and then the main process
terminates zero.  This means that the test seems to pass if you look at its=
 $?,
but the harness sees it as a failure because its total output does include
failures (and non-increasing test numbers).

Adding Test2::IPC does not help.  The issue is not synchronising test count=
ers,
or the like.  It's that I don't want the subprocess's exit to cause a subte=
st
exit in the parent process.  I think that's right: the only process that sh=
ould
turn exceptions into subtest fails is the process that created the subtest.

a) am I wrong?
b) advice on how to procede?

--=20
rjbs

--PEIAKu/WMn1b1Hv9
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJYUByMAAoJEOYby6cMccU5yfwH/2mimn95AJn6ghmPlzQMNZnr
B/N3dzXniYSwHOwA0YJBGji525YPz/W+i9gpxi/GCcwz5+tFA/NgguN2BwsUA8Q2
LvKhJDp7xRKLHUVhXkwxqRuipmUHXtlOZuzRcqtg4DaSi2TxZPPs+pyz9mKhG/gD
W+RKiZsqVxb/PwouCgIVKip9/ZwVvT1f0zHcdSOR7z6keQ2QxCWdBngUBn3q7Ugo
uiRsKTIwAJsQYHtr+n2Z7+vZze/UA43dchRrD+gJyFHleGVAHFam3pekpdwA8lgg
vYwR/HbzcPBiKqzfWn3eiz3BqwpuklPVlUGW810wzh5ngCbSJTK0CdgHYCN5N44=
=kQoq
-----END PGP SIGNATURE-----

--PEIAKu/WMn1b1Hv9--