Re: Test::Buildder v. Test2 subtests
[email protected] (Chad Granum) Tue, 13 Dec 2016 08:09:29 -0800
| Newsgroups | perl.qa |
|---|---|
| Message-ID | <CAJFr3ktWv4VvPN_E52s_2Zv-HtaB_juLN9rRYsbhRZPAukG3hg@mail.gmail.com> |
--f403045e257ef0491505438c7124 Content-Type: text/plain; charset=UTF-8 I will have to look into this when I have more time. On Tue, Dec 13, 2016 at 8:06 AM, Ricardo Signes <[email protected]> wrote: > I've hit a nasty (to me) difference between Test2::API::run_subtest and > Test::Builder::subtest. Shout out to Matthew Horsfall for helping > localize the > problem. Here's my trivial reproducer: > > #!perl > use strict; > use warnings; > use Test::More; > use Test2::API; > > my $code = sub { > if (fork) { > wait; > cmp_ok($?, '!=', 0, "subprocess died"); > } else { > die "123"; > }; > }; > > my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest; > $call->(test => $code); > done_testing; > > When running with Test::More... > > ~$ perl foo 1 > # Subtest: test > 123 at foo line 12. > ok 1 - subprocess died > 1..1 > ok 1 - test > 1..1 > > The die in the subprocess causes the child process to exit nonzero and the > tests run normally. Great! This is how my tests work. When I switch them > to Test2::API::run_subtest, though, I get this: > > ~$ perl foo 0 > # test > not ok 1 - test > # Failed test 'test' > # at foo line 17. > # Caught exception in subtest: 123 at foo line 12. > 1..1 > not ok 1 - subprocess died > # Failed test 'subprocess died' > # at foo line 10. > # got: 0 > # expected: anything else > 1..1 > not ok 1 - test > # Failed test 'test' > # at foo line 17. > 1..1 > # Looks like you failed 1 test of 1. > > In the forked code, the die causes the subtest to end in failure *in the > fork* > meaning we get a repeat result for test 1: one from the child, one from the > parent. Also, the child is now exiting zero instead of nonzero, despite > the > fact that it threw an exception. > > It gets worse [for me]! Above, I had a cmp_ok on $? that failed, so I had > duplicate *not ok* results for test 1. If I replace that cmp_ok with a > pass, I > get duplicate *conflicting* results for test 1... and then the main process > terminates zero. This means that the test seems to pass if you look at > its $?, > but the harness sees it as a failure because its total output does include > failures (and non-increasing test numbers). > > Adding Test2::IPC does not help. The issue is not synchronising test > counters, > or the like. It's that I don't want the subprocess's exit to cause a > subtest > exit in the parent process. I think that's right: the only process that > should > turn exceptions into subtest fails is the process that created the subtest. > > a) am I wrong? > b) advice on how to procede? > > -- > rjbs > --f403045e257ef0491505438c7124 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I will have to look into this when I have more time.</div>= <div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Dec 13, 2= 016 at 8:06 AM, Ricardo Signes <span dir=3D"ltr"><<a href=3D"mailto:perl= [email protected]" target=3D"_blank">[email protected]</a>></s= pan> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex= ;border-left:1px #ccc solid;padding-left:1ex">I've hit a nasty (to me) = difference between Test2::API::run_subtest and<br> Test::Builder::subtest.=C2=A0 Shout out to Matthew Horsfall for helping loc= alize the<br> problem.=C2=A0 Here's my trivial reproducer:<br> <br> =C2=A0 #!perl<br> =C2=A0 use strict;<br> =C2=A0 use warnings;<br> =C2=A0 use Test::More;<br> =C2=A0 use Test2::API;<br> <br> =C2=A0 my $code =3D sub {<br> =C2=A0 =C2=A0 if (fork) {<br> =C2=A0 =C2=A0 =C2=A0 wait;<br> =C2=A0 =C2=A0 =C2=A0 cmp_ok($?, '!=3D', 0, "subprocess died&qu= ot;);<br> =C2=A0 =C2=A0 } else {<br> =C2=A0 =C2=A0 =C2=A0 die "123";<br> =C2=A0 =C2=A0 };<br> =C2=A0 };<br> <br> =C2=A0 my $call =3D $ARGV[0] ? \&Test::More::subtest : \&Test2::API= ::run_subtest;<br> =C2=A0 $call->(test =3D> $code);<br> =C2=A0 done_testing;<br> <br> When running with Test::More...<br> <br> =C2=A0 ~$ perl foo 1<br> =C2=A0 # Subtest: test<br> =C2=A0 123 at foo line 12.<br> =C2=A0 =C2=A0 =C2=A0 ok 1 - subprocess died<br> =C2=A0 =C2=A0 =C2=A0 1..1<br> =C2=A0 ok 1 - test<br> =C2=A0 1..1<br> <br> The die in the subprocess causes the child process to exit nonzero and the<= br> tests run normally.=C2=A0 Great!=C2=A0 This is how my tests work.=C2=A0 Whe= n I switch them<br> to Test2::API::run_subtest, though, I get this:<br> <br> =C2=A0 ~$ perl foo 0<br> =C2=A0 # test<br> =C2=A0 not ok 1 - test<br> =C2=A0 # Failed test 'test'<br> =C2=A0 # at foo line 17.<br> =C2=A0 # Caught exception in subtest: 123 at foo line 12.<br> =C2=A0 1..1<br> =C2=A0 =C2=A0 =C2=A0 not ok 1 - subprocess died<br> =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0Failed test 'subprocess died'<br= > =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0at foo line 10.<br> =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 got: 0<br> =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0 =C2=A0expected: anything else<br> =C2=A0 =C2=A0 =C2=A0 1..1<br> =C2=A0 not ok 1 - test<br> =C2=A0 # Failed test 'test'<br> =C2=A0 # at foo line 17.<br> =C2=A0 1..1<br> =C2=A0 # Looks like you failed 1 test of 1.<br> <br> In the forked code, the die causes the subtest to end in failure *in the fo= rk*<br> meaning we get a repeat result for test 1: one from the child, one from the= <br> parent.=C2=A0 Also, the child is now exiting zero instead of nonzero, despi= te the<br> fact that it threw an exception.<br> <br> It gets worse [for me]!=C2=A0 Above, I had a cmp_ok on $? that failed, so I= had<br> duplicate *not ok* results for test 1.=C2=A0 If I replace that cmp_ok with = a pass, I<br> get duplicate *conflicting* results for test 1... and then the main process= <br> terminates zero.=C2=A0 This means that the test seems to pass if you look a= t its $?,<br> but the harness sees it as a failure because its total output does include<= br> failures (and non-increasing test numbers).<br> <br> Adding Test2::IPC does not help.=C2=A0 The issue is not synchronising test = counters,<br> or the like.=C2=A0 It's that I don't want the subprocess's exit= to cause a subtest<br> exit in the parent process.=C2=A0 I think that's right: the only proces= s that should<br> turn exceptions into subtest fails is the process that created the subtest.= <br> <br> a) am I wrong?<br> b) advice on how to procede?<br> <span class=3D"HOEnZb"><font color=3D"#888888"><br> --<br> rjbs<br> </font></span></blockquote></div><br></div> --f403045e257ef0491505438c7124--