Re: Bugs reported because of failing tests more likely to be reopened?

--001a11c3556e537af304ebb9520c
Content-Type: text/plain; charset=ISO-8859-1

Just a thought, but could it be that some bugs with the label TEST fail not
because of a problem in the code but because of intermittent problems in
the test environment? If you have QA tests that do something with files,
for example, then you might randomly run into problems with file locking
when running them under Windows depending on what other processes happen to
be running right then.


On Thu, Nov 21, 2013 at 5:54 PM, Rodrigo Rocha Gomes e Souza <
[email protected]> wrote:

> Hello again,
>
> I was doing some statistical analyses with NetBeans Platform's bug reports
> and concluded that...
>
> ... bugs related to failing, unfinishing or errors in the test (i.e., bugs
> with keyword TEST) are 2.5x more likely to be reopened after being fixed.
>
> Do you have any theories that explain this result?
>
> The result was surprising for me, because I thought it would be the other
> way around: bugs reported because of a failing test would be *less* likely
> to be reopened because they should be easier to reproduce. After all, in
> this case the developer himself can verify if the fix was appropriate just
> by running the failing test. What do you think?
>
> []s
> Rodrigo
> Federal University of Bahia
>



-- 
Tom Wheeler
http://www.tomwheeler.com/

--001a11c3556e537af304ebb9520c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Just a thought, but could it be that some bugs with the la=
bel TEST fail not because of a problem in the code but because of intermitt=
ent problems in the test environment? If you have QA tests that do somethin=
g with files, for example, then you might randomly run into problems with f=
ile locking when running them under Windows depending on what other process=
es happen to be running right then.<br>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Thu,=
 Nov 21, 2013 at 5:54 PM, Rodrigo Rocha Gomes e Souza <span dir=3D"ltr">&lt=
;<a href=3D"mailto:[email protected]" target=3D"_blank">rodrigorgs@gmail=
.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hello again,<div><br></div>=
<div>I was doing some statistical analyses with NetBeans Platform&#39;s bug=
 reports and concluded that...</div>
<div><br></div><div>... bugs related to failing, unfinishing or errors in t=
he test (i.e., bugs with keyword TEST) are 2.5x more likely to be reopened =
after being fixed.</div>

<div><br></div><div>Do you have any theories that explain this result?</div=
><div><br></div><div>The result was surprising for me, because I thought it=
 would be the other way around: bugs reported because of a failing test wou=
ld be *less* likely to be reopened because they should be easier to reprodu=
ce. After all, in this case the developer himself can verify if the fix was=
 appropriate just by running the failing test. What do you think?</div>


<div><br></div><div>[]s</div><div>Rodrigo</div><div>Federal University of B=
ahia</div></div>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Tom Wheeler<br><a href=
=3D"http://www.tomwheeler.com/">http://www.tomwheeler.com/</a>
</div>

--001a11c3556e537af304ebb9520c--