RE: ES3 update 8 has severe problems with "pvm"

There was a comment posted to bug #191841 yesterday
(https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=191841#c31).  This
reply was from Tom Kincaid, who is the director of the Quality Engineering
organization at Red Hat.  I'm reposting it here.  It gives some insights
into what happened in this case and what additional steps Red Hat is
taking in future testing.

Comment #31 From Tom Kincaid ([email protected]) on 2006-07-27 14:14 EST

This message is in reference to Red Hat Bugzilla 191841. The synopsis of
this bug is that a package on RHN failed to install properly via up2date.
Depending on how users run up2date, this can leave systems in an
inconsistent state.

In the spirit of transparency, included in this post are the following
items:

1) An overview of what our current test plan calls for relative to up2date
package testing.

2) What went wrong in this case.

3) What we are doing to keep the problem from occurring again.



1- Current test plan

a) Each package is tested individually, to verify that it can be updated
from the previous RHEL update.

b) We test that a virgin RHEL X.uY release can be completely updated to a
RHEL X.uY+1 via up2date.

c) Developers must execute a static analysis tool on each package that is
going to be shipped via RHN. This tool often uncovers and helps us resolve
problems with packages that cause up2date to fail.

I have been at Red Hat over 18 months. The above test plan steps have
caught numerous upgrade issue like this bug before they could ever get
close to making it out of the building. In addition, as far as I know this
is the first problem of this nature that has made it out of the building
in recent years.

2 - What went wrong in this case.

-We got conflicting test results from items 1a and 1b.

-What created some confusion for the testing organization was that the pvm
Errata had shipped as a FastTrack errata in early May and no external
parties were reporting this bug and we had positive up2date testing
results for this Errata. We made a judgment call that the tests from step
1a were the correct results and the tests from step 1b were incorrect.

-The conclusion that the test results from step 1a were correct and step
1b were not correct was a failure in judgment on our part. I understand
that it appears obvious that we should have drilled down on the failures
as result of executing step 1b. We are tightening up our release criteria
and test plans to make sure this does not happen again.


3 - Here is what we are doing to fix things going forward:

-We are putting a check in the tool mentioned in Step 1c that will catch
the specific issue with this rpm.

-No package will be released with out positive results from both steps 1a
and 1b.

-We are fixing the issues in step 1b that resulted in the false positive
result. Specifically that the system was not a completely virgin U7 system
when the test was performed. It had some additional packages on it.





--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list