Re: Test::Buildder v. Test2 subtests
[email protected] (Chad Granum) Tue, 13 Dec 2016 08:13:56 -0800
| Newsgroups | perl.qa |
|---|---|
| Message-ID | <CAJFr3kubayX3ei4ZZi_dF1O8iDciEnzF7CZyxiOf93v+bvh2PA@mail.gmail.com> |
--94eb2c03a7dadea15b05438c8137 Content-Type: text/plain; charset=UTF-8 Actually at first glance a possible problem is this: You are using 2 functions that are not intended to be interchangeable. Test::More::subtest() is a tool intended to run a subtest. Test2::API::run_subtest() is a helper intended to write tools like 'subtest()'. I would like to see the results when you use Test2::Tools::Subtest::subtest_streamed() which is the actual equivalent to Test::More::subtest(). -Chad On Tue, Dec 13, 2016 at 8:09 AM, Chad Granum <[email protected]> wrote: > I will have to look into this when I have more time. > > On Tue, Dec 13, 2016 at 8:06 AM, Ricardo Signes <[email protected]> > wrote: > >> I've hit a nasty (to me) difference between Test2::API::run_subtest and >> Test::Builder::subtest. Shout out to Matthew Horsfall for helping >> localize the >> problem. Here's my trivial reproducer: >> >> #!perl >> use strict; >> use warnings; >> use Test::More; >> use Test2::API; >> >> my $code = sub { >> if (fork) { >> wait; >> cmp_ok($?, '!=', 0, "subprocess died"); >> } else { >> die "123"; >> }; >> }; >> >> my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest; >> $call->(test => $code); >> done_testing; >> >> When running with Test::More... >> >> ~$ perl foo 1 >> # Subtest: test >> 123 at foo line 12. >> ok 1 - subprocess died >> 1..1 >> ok 1 - test >> 1..1 >> >> The die in the subprocess causes the child process to exit nonzero and the >> tests run normally. Great! This is how my tests work. When I switch >> them >> to Test2::API::run_subtest, though, I get this: >> >> ~$ perl foo 0 >> # test >> not ok 1 - test >> # Failed test 'test' >> # at foo line 17. >> # Caught exception in subtest: 123 at foo line 12. >> 1..1 >> not ok 1 - subprocess died >> # Failed test 'subprocess died' >> # at foo line 10. >> # got: 0 >> # expected: anything else >> 1..1 >> not ok 1 - test >> # Failed test 'test' >> # at foo line 17. >> 1..1 >> # Looks like you failed 1 test of 1. >> >> In the forked code, the die causes the subtest to end in failure *in the >> fork* >> meaning we get a repeat result for test 1: one from the child, one from >> the >> parent. Also, the child is now exiting zero instead of nonzero, despite >> the >> fact that it threw an exception. >> >> It gets worse [for me]! Above, I had a cmp_ok on $? that failed, so I had >> duplicate *not ok* results for test 1. If I replace that cmp_ok with a >> pass, I >> get duplicate *conflicting* results for test 1... and then the main >> process >> terminates zero. This means that the test seems to pass if you look at >> its $?, >> but the harness sees it as a failure because its total output does include >> failures (and non-increasing test numbers). >> >> Adding Test2::IPC does not help. The issue is not synchronising test >> counters, >> or the like. It's that I don't want the subprocess's exit to cause a >> subtest >> exit in the parent process. I think that's right: the only process that >> should >> turn exceptions into subtest fails is the process that created the >> subtest. >> >> a) am I wrong? >> b) advice on how to procede? >> >> -- >> rjbs >> > > --94eb2c03a7dadea15b05438c8137 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Actually at first glance a possible problem is this: You a= re using 2 functions that are not intended to be interchangeable.<div><br><= /div><div>Test::More::subtest() is a tool intended to run a subtest. Test2:= :API::run_subtest() is a helper intended to write tools like 'subtest()= '.</div><div><br></div><div>I would like to see the results when you us= e Test2::Tools::Subtest::subtest_streamed() which is the actual equivalent = to Test::More::subtest().</div><div><br></div><div>-Chad</div></div><div cl= ass=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Dec 13, 2016 at = 8:09 AM, Chad Granum <span dir=3D"ltr"><<a href=3D"mailto:exodist7@gmail= .com" target=3D"_blank">[email protected]</a>></span> wrote:<br><blockq= uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div dir=3D"ltr">I will have to look into this when= I have more time.</div><div class=3D"HOEnZb"><div class=3D"h5"><div class= =3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Dec 13, 2016 at 8:0= 6 AM, Ricardo Signes <span dir=3D"ltr"><<a href=3D"mailto:[email protected]= anxome.org" target=3D"_blank">[email protected]</a>></span> wrote= :<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-le= ft:1px #ccc solid;padding-left:1ex">I've hit a nasty (to me) difference= between Test2::API::run_subtest and<br> Test::Builder::subtest.=C2=A0 Shout out to Matthew Horsfall for helping loc= alize the<br> problem.=C2=A0 Here's my trivial reproducer:<br> <br> =C2=A0 #!perl<br> =C2=A0 use strict;<br> =C2=A0 use warnings;<br> =C2=A0 use Test::More;<br> =C2=A0 use Test2::API;<br> <br> =C2=A0 my $code =3D sub {<br> =C2=A0 =C2=A0 if (fork) {<br> =C2=A0 =C2=A0 =C2=A0 wait;<br> =C2=A0 =C2=A0 =C2=A0 cmp_ok($?, '!=3D', 0, "subprocess died&qu= ot;);<br> =C2=A0 =C2=A0 } else {<br> =C2=A0 =C2=A0 =C2=A0 die "123";<br> =C2=A0 =C2=A0 };<br> =C2=A0 };<br> <br> =C2=A0 my $call =3D $ARGV[0] ? \&Test::More::subtest : \&Test2::API= ::run_subtest;<br> =C2=A0 $call->(test =3D> $code);<br> =C2=A0 done_testing;<br> <br> When running with Test::More...<br> <br> =C2=A0 ~$ perl foo 1<br> =C2=A0 # Subtest: test<br> =C2=A0 123 at foo line 12.<br> =C2=A0 =C2=A0 =C2=A0 ok 1 - subprocess died<br> =C2=A0 =C2=A0 =C2=A0 1..1<br> =C2=A0 ok 1 - test<br> =C2=A0 1..1<br> <br> The die in the subprocess causes the child process to exit nonzero and the<= br> tests run normally.=C2=A0 Great!=C2=A0 This is how my tests work.=C2=A0 Whe= n I switch them<br> to Test2::API::run_subtest, though, I get this:<br> <br> =C2=A0 ~$ perl foo 0<br> =C2=A0 # test<br> =C2=A0 not ok 1 - test<br> =C2=A0 # Failed test 'test'<br> =C2=A0 # at foo line 17.<br> =C2=A0 # Caught exception in subtest: 123 at foo line 12.<br> =C2=A0 1..1<br> =C2=A0 =C2=A0 =C2=A0 not ok 1 - subprocess died<br> =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0Failed test 'subprocess died'<br= > =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0at foo line 10.<br> =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 got: 0<br> =C2=A0 =C2=A0 =C2=A0 #=C2=A0 =C2=A0 =C2=A0expected: anything else<br> =C2=A0 =C2=A0 =C2=A0 1..1<br> =C2=A0 not ok 1 - test<br> =C2=A0 # Failed test 'test'<br> =C2=A0 # at foo line 17.<br> =C2=A0 1..1<br> =C2=A0 # Looks like you failed 1 test of 1.<br> <br> In the forked code, the die causes the subtest to end in failure *in the fo= rk*<br> meaning we get a repeat result for test 1: one from the child, one from the= <br> parent.=C2=A0 Also, the child is now exiting zero instead of nonzero, despi= te the<br> fact that it threw an exception.<br> <br> It gets worse [for me]!=C2=A0 Above, I had a cmp_ok on $? that failed, so I= had<br> duplicate *not ok* results for test 1.=C2=A0 If I replace that cmp_ok with = a pass, I<br> get duplicate *conflicting* results for test 1... and then the main process= <br> terminates zero.=C2=A0 This means that the test seems to pass if you look a= t its $?,<br> but the harness sees it as a failure because its total output does include<= br> failures (and non-increasing test numbers).<br> <br> Adding Test2::IPC does not help.=C2=A0 The issue is not synchronising test = counters,<br> or the like.=C2=A0 It's that I don't want the subprocess's exit= to cause a subtest<br> exit in the parent process.=C2=A0 I think that's right: the only proces= s that should<br> turn exceptions into subtest fails is the process that created the subtest.= <br> <br> a) am I wrong?<br> b) advice on how to procede?<br> <span class=3D"m_-312982469230195390HOEnZb"><font color=3D"#888888"><br> --<br> rjbs<br> </font></span></blockquote></div><br></div> </div></div></blockquote></div><br></div> --94eb2c03a7dadea15b05438c8137--