Re: test/subtest flow control with exceptions
[email protected] (Chad Granum) Wed, 30 Nov 2016 07:30:38 -0800
| Newsgroups | perl.qa |
|---|---|
| Message-ID | <CAJFr3ks7rKnwV6gAFyBy=sEH+OFH3An9tEhYoFbmwMr5-+gLgw@mail.gmail.com> |
--f403045e257e1860690542866311 Content-Type: text/plain; charset=UTF-8 I don't have much comment on the functionality you want, seems reasonable enough... I do have implementation commentary however: * You should not be obtaining a context inside your subtest (specifically line 18 https://github.com/rjbs/Test-Abortable/blob/master/lib/Test/Abortable.pm#L18). Obtaining that context means that tests run inside your eval are likely to report errors to the wrong files+lines. If you need a context that is within the subtest you should obtain it after your eval. * https://github.com/rjbs/Test-Abortable/blob/master/lib/Test/Abortable.pm#L30 this should probably be $ctx->throw() which is essentially a die, but it "aborts" the context so that it is released properly (not the die you use prevents context->release from being called). Though on second thought, throw() will append a file+line number to your exception, so you should probably just add a $ctx->release right before that die. (and on further reading your second sub asks if release should be done, the answer is yes. -Chad On Tue, Nov 29, 2016 at 5:52 PM, Ricardo Signes <[email protected]> wrote: > Often, I have a test like this: > > subtest "do things with an api" => sub { > my $result = $api_client->do_first_thing; > > is( > $result->documents->first->title, > "The Best Thing", > ); > > ... > }; > > Sometimes, the result comes back with zero documents. ->first throws an > exception and then my whole test program comes crashing down and it's > miserable. For a while, I've been meaning to make it possible for some > exceptions to be recognized by my test programs as instructions to emit a > failure, stop this subtest, and move on. > > It's important to note that I'm talking, in the code above, about an > exception > that would be thrown by ($result->documents) when ->first is called on it. > This kind of flow control eliminates needing to write: > > my $result = $api_client->do_first_thing; > my $docs = $result->documents; > fail("no docs"), return unless $docs->has_entries; > my $first = $docs->first; > fail("no title"), return unless $first->has_title; > > is(...); > > In my case, I'm working with an API client that's specifically designed to > be > used for testing, so this kind of loose coupling between thrown exceptions > and > the test code is a good fit. Not every exception should be caught this > way. > Truly unexpected ones should still die. > > So, I've written something to do this, and I'm sharing it here before going > further with it. In my code, if an exception is meant to be caught and > used as > a local abort instruction, it has a method called as_test_abort_events. > This > method returns a reference to an array of Test2 event descriptions. For > example, maybe: > > sub as_test_abort_events { > return [ > [ Ok => (pass => 0, name => "no documents, but ->first called") ], > [ Diag => (message => "collection state: ....") ], > ]; > } > > This method is easy to add to any exception you want, and you don't need to > change your exception class hierarchy in any way. > > Next, you need something to run the tests and look for these exceptions > being > thrown. I have written a library for this, called Test::Abortable. > > https://github.com/rjbs/Test-Abortable > > Test::Abortable provides two subroutines: subtest and testeval. subtest > acts > just like Test::More's subtest, but catches abort exceptions, emits their > events, and returns normally. (I've also updated my library > Test::Routine, in > a branch, to behave this way, as I use it in place of subtest for many > things.) > > testeval acts like eval, but only catches abort exceptions. > > ok(1); > testeval { > ok(2); > this_throws_an_abort; > ok(3); > }; > ok(4); > > This ends up emitting ok 1, ok 2, whatever the abort wants, and ok 4. > testeval > returns the return value of the code block if it succeeds. If it fails > due to > abort, it returns false, emits the abort events, and puts the abort in > $@. If > it fails because of any other exception, the exception is re-thrown. > > Let me know if you have any thoughts before I begin using this in anger. > :-) > > -- > rjbs > --f403045e257e1860690542866311 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I don't have much comment on the functionality you wan= t, seems reasonable enough...<div><br></div><div>I do have implementation c= ommentary however:</div><div><br></div><div>=C2=A0* You should not be obtai= ning a context inside your subtest (specifically line 18=C2=A0<a href=3D"ht= tps://github.com/rjbs/Test-Abortable/blob/master/lib/Test/Abortable.pm#L18"= >https://github.com/rjbs/Test-Abortable/blob/master/lib/Test/Abortable.pm#L= 18</a>). Obtaining that context means that tests run inside your eval are l= ikely to report errors to the wrong files+lines. If you need a context that= is within the subtest you should obtain it after your eval.<br></div><div>= <br></div><div>=C2=A0*=C2=A0<a href=3D"https://github.com/rjbs/Test-Abortab= le/blob/master/lib/Test/Abortable.pm#L30">https://github.com/rjbs/Test-Abor= table/blob/master/lib/Test/Abortable.pm#L30</a> =C2=A0this should probably = be $ctx->throw() which is essentially a die, but it "aborts" t= he context so that it is released properly (not the die you use prevents co= ntext->release from being called). Though on second thought, throw() wil= l append a file+line number to your exception, so you should probably just = add a $ctx->release right before that die. (and on further reading your = second sub asks if release should be done, the answer is yes.</div><div><br= ></div><div>-Chad</div><div><br></div><div><br></div><div><br></div><div><b= r></div><div><br></div><div><br></div></div><div class=3D"gmail_extra"><br>= <div class=3D"gmail_quote">On Tue, Nov 29, 2016 at 5:52 PM, Ricardo Signes = <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D= "_blank">[email protected]</a>></span> wrote:<br><blockquote clas= s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad= ding-left:1ex">Often, I have a test like this:<br> <br> =C2=A0 subtest "do things with an api" =3D> sub {<br> =C2=A0 =C2=A0 my $result =3D $api_client->do_first_thing;<br> <br> =C2=A0 =C2=A0 is(<br> =C2=A0 =C2=A0 =C2=A0 $result->documents->first-><wbr>title,<br> =C2=A0 =C2=A0 =C2=A0 "The Best Thing",<br> =C2=A0 =C2=A0 );<br> <br> =C2=A0 =C2=A0 ...<br> =C2=A0 };<br> <br> Sometimes, the result comes back with zero documents.=C2=A0 ->first thro= ws an<br> exception and then my whole test program comes crashing down and it's<b= r> miserable.=C2=A0 For a while, I've been meaning to make it possible for= some<br> exceptions to be recognized by my test programs as instructions to emit a<b= r> failure, stop this subtest, and move on.<br> <br> It's important to note that I'm talking, in the code above, about a= n exception<br> that would be thrown by ($result->documents) when ->first is called o= n it.<br> This kind of flow control eliminates needing to write:<br> <br> =C2=A0 my $result =3D $api_client->do_first_thing;<br> =C2=A0 my $docs=C2=A0 =C2=A0=3D $result->documents;<br> =C2=A0 fail("no docs"), return unless $docs->has_entries;<br> =C2=A0 my $first=C2=A0 =3D $docs->first;<br> =C2=A0 fail("no title"), return unless $first->has_title;<br> <br> =C2=A0 is(...);<br> <br> In my case, I'm working with an API client that's specifically desi= gned to be<br> used for testing, so this kind of loose coupling between thrown exceptions = and<br> the test code is a good fit.=C2=A0 Not every exception should be caught thi= s way.<br> Truly unexpected ones should still die.<br> <br> So, I've written something to do this, and I'm sharing it here befo= re going<br> further with it.=C2=A0 In my code, if an exception is meant to be caught an= d used as<br> a local abort instruction, it has a method called as_test_abort_events.=C2= =A0 This<br> method returns a reference to an array of Test2 event descriptions.=C2=A0 F= or<br> example, maybe:<br> <br> =C2=A0 sub as_test_abort_events {<br> =C2=A0 =C2=A0 return [<br> =C2=A0 =C2=A0 =C2=A0 [ Ok=C2=A0 =C2=A0=3D> (pass =3D> 0, name =3D>= "no documents, but ->first called") ],<br> =C2=A0 =C2=A0 =C2=A0 [ Diag =3D> (message =3D> "collection state= : ....") ],<br> =C2=A0 =C2=A0 ];<br> =C2=A0 }<br> <br> This method is easy to add to any exception you want, and you don't nee= d to<br> change your exception class hierarchy in any way.<br> <br> Next, you need something to run the tests and look for these exceptions bei= ng<br> thrown.=C2=A0 I have written a library for this, called Test::Abortable.<br= > <br> =C2=A0 <a href=3D"https://github.com/rjbs/Test-Abortable" rel=3D"noreferrer= " target=3D"_blank">https://github.com/rjbs/Test-<wbr>Abortable</a><br> <br> Test::Abortable provides two subroutines: subtest and testeval.=C2=A0 subte= st acts<br> just like Test::More's subtest, but catches abort exceptions, emits the= ir<br> events, and returns normally.=C2=A0 (I've also updated my library Test:= :Routine, in<br> a branch, to behave this way, as I use it in place of subtest for many thin= gs.)<br> <br> testeval acts like eval, but only catches abort exceptions.<br> <br> =C2=A0 ok(1);<br> =C2=A0 testeval {<br> =C2=A0 =C2=A0 ok(2);<br> =C2=A0 =C2=A0 this_throws_an_abort;<br> =C2=A0 =C2=A0 ok(3);<br> =C2=A0 };<br> =C2=A0 ok(4);<br> <br> This ends up emitting ok 1, ok 2, whatever the abort wants, and ok 4.=C2=A0= testeval<br> returns the return value of the code block if it succeeds.=C2=A0 If it fail= s due to<br> abort, it returns false, emits the abort events, and puts the abort in $@.= =C2=A0 If<br> it fails because of any other exception, the exception is re-thrown.<br> <br> Let me know if you have any thoughts before I begin using this in anger. :-= )<br> <span class=3D"HOEnZb"><font color=3D"#888888"><br> --<br> rjbs<br> </font></span></blockquote></div><br></div> --f403045e257e1860690542866311--