Re: Test2/Test::Builder release plan

--001a11427d740f0d88052fe9c68b
Content-Type: text/plain; charset=UTF-8

>
>
> I think 1 is a good idea, but I have some reservations about the 2 (and
> thus 4). Is it really advantageous to switch over everyone to Test2 today?
>
> I think Test2 has major benefits for some people (it makes new things
> possible), but it also has major disadvantages for others (it breaks
> stuff); both have unknown upper bounds. And to be honest, for the vast
> majority I don't feel like it will matter; they don't need (much) more than
> Test::More.
>
> Wouldn't it make more sense to make this opt-in?
>
> Leon
>
>
The main benefits of switch as opposed to opt-in are these:

 * Only maintaining one system. Yes we still need to maintain Test::Builder
and Test2, but when combined it reduces the cognitive overhead, and the
disconnect between the 2.
 * The code to switch includes bugfixes that cannot be easily/sanely
backported to old test builder
    - Threads + Subtests
    - Forking + Subtests
    - False pass if parent finishes before child thread/process where child
then has a failure
    - Probably more
 * People will not have to be worried about something downstream switching
and thus forcing them to switch unknowingly (cause the switch happens once,
for everyone)

The problems of making it opt-in are the same problems you always get when
you maintain 2 separate but similar things. Things become desynchronized
between them. People have to write their stuff to support both, or simply
make their code die with a message telling you not to use one or the other.
If we do not combine these then we put the burden of supporting both on
everyone else. If however the next Test::Builder used Test2 then the burden
is on me and any other Test2/Test::Builder maintainer. Yes there is still
the toolchain burden of making sure stuff works on new and old
Test::Builder, but toolchain is already its own beast.


I think Test2 has major benefits for some people (it makes new things
> possible), but it also has major disadvantages for others (it breaks
> stuff); both have unknown upper bounds. And to be honest, for the vast
> majority I don't feel like it will matter; they don't need (much) more than
> Test::More.


The main disadvantage is that it "breaks stuff". I do not want to
trivialize this, it is a huge concern. And you are correct, we do not know
the upper bound. What we do know though is that cpan shows us. Currently
the breakage on cpan is very small, limited to only a handful of modules,
many of which already have pending patches, or people willing to pick them
up. And most, if not all of what breaks is people bypassing Test::Builders
public API's and mucking about directly with internals. Even in these cases
hoops have been jumped though to support as many of these bad practices as
technically possible.

I do not want to dismiss or trivialize the disadvantages, I want to be
clear on that. However I consider the advantages to have significantly more
weight here. I think the burden on cpan of supporting a split system here
is a much higher risk. If the author of Test::Moose for instance decides to
switch to Test2, then any test that consumes it switches automatically. If
there is a conflict in any of the used tools then people have breakage, and
it is not as predictable. If we switch all at once though, then a module
author deciding to switch simply needs to bump up the minimum Test::Builder
version required, anything that works on that higher version *should* be
set.

-Chad

--001a11427d740f0d88052fe9c68b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;paddi=
ng-left:1ex"><div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmai=
l_quote"><span class=3D""><div>=C2=A0<br></div></span><div>I think 1 is a g=
ood idea, but I have some reservations about the 2 (and thus 4). Is it real=
ly advantageous to switch over everyone to Test2 today?<br><br></div><div>I=
 think Test2 has major benefits for some people (it makes new things possib=
le), but it also has major disadvantages for others (it breaks stuff); both=
 have unknown upper bounds. And to be honest, for the vast majority I don&#=
39;t feel like it will matter; they don&#39;t need (much) more than Test::M=
ore.<br></div><div><br></div><div>Wouldn&#39;t it make more sense to make t=
his opt-in?<span class=3D""><font color=3D"#888888"><br><br></font></span><=
/div><span class=3D""><font color=3D"#888888"><div>Leon<br><br></div></font=
></span></div></div></div>
</blockquote></div><br></div><div class=3D"gmail_extra">The main benefits o=
f switch as opposed to opt-in are these:</div><div class=3D"gmail_extra"><b=
r></div><div class=3D"gmail_extra">=C2=A0* Only maintaining one system. Yes=
 we still need to maintain Test::Builder and Test2, but when combined it re=
duces the cognitive overhead, and the disconnect between the 2.</div><div c=
lass=3D"gmail_extra">=C2=A0* The code to switch includes bugfixes that cann=
ot be easily/sanely backported to old test builder</div><div class=3D"gmail=
_extra">=C2=A0 =C2=A0 - Threads + Subtests</div><div class=3D"gmail_extra">=
=C2=A0 =C2=A0 - Forking + Subtests</div><div class=3D"gmail_extra">=C2=A0 =
=C2=A0 - False pass if parent finishes before child thread/process where ch=
ild then has a failure</div><div class=3D"gmail_extra">=C2=A0 =C2=A0 - Prob=
ably more</div><div class=3D"gmail_extra">=C2=A0* People will not have to b=
e worried about something downstream switching and thus forcing them to swi=
tch unknowingly (cause the switch happens once, for everyone)</div><div cla=
ss=3D"gmail_extra"><br></div><div class=3D"gmail_extra">The problems of mak=
ing it opt-in are the same problems you always get when you maintain 2 sepa=
rate but similar things. Things become desynchronized between them. People =
have to write their stuff to support both, or simply make their code die wi=
th a message telling you not to use one or the other. If we do not combine =
these then we put the burden of supporting both on everyone else. If howeve=
r the next Test::Builder used Test2 then the burden is on me and any other =
Test2/Test::Builder maintainer. Yes there is still the toolchain burden of =
making sure stuff works on new and old Test::Builder, but toolchain is alre=
ady its own beast.<br></div><div class=3D"gmail_extra"><br></div><div class=
=3D"gmail_extra"><br></div><div class=3D"gmail_extra"><blockquote class=3D"=
gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border=
-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I th=
ink Test2 has major benefits for some people (it makes new things possible)=
, but it also has major disadvantages for others (it breaks stuff); both ha=
ve unknown upper bounds. And to be honest, for the vast majority I don&#39;=
t feel like it will matter; they don&#39;t need (much) more than Test::More=
.</blockquote><div><br></div><div>The main disadvantage is that it &quot;br=
eaks stuff&quot;. I do not want to trivialize this, it is a huge concern. A=
nd you are correct, we do not know the upper bound. What we do know though =
is that cpan shows us. Currently the breakage on cpan is very small, limite=
d to only a handful of modules, many of which already have pending patches,=
 or people willing to pick them up. And most, if not all of what breaks is =
people bypassing Test::Builders public API&#39;s and mucking about directly=
 with internals. Even in these cases hoops have been jumped though to suppo=
rt as many of these bad practices as technically possible.</div><div><br></=
div><div>I do not want to dismiss or trivialize the disadvantages, I want t=
o be clear on that. However I consider the advantages to have significantly=
 more weight here. I think the burden on cpan of supporting a split system =
here is a much higher risk. If the author of Test::Moose for instance decid=
es to switch to Test2, then any test that consumes it switches automaticall=
y. If there is a conflict in any of the used tools then people have breakag=
e, and it is not as predictable. If we switch all at once though, then a mo=
dule author deciding to switch simply needs to bump up the minimum Test::Bu=
ilder version required, anything that works on that higher version *should*=
 be set.</div><div><br></div><div>-Chad</div></div></div>

--001a11427d740f0d88052fe9c68b--