RFC: Proper Addition Of New Tests
[email protected] (Michael G Schwern)
| Newsgroups | perl.bootstrap,perl.qa |
|---|---|
| Message-ID | <[email protected]> |
In One Sentence --------------- It will be checked that new tests fail (they find the bug) before the fix is applied. Synopsis -------- Its is often forgotten that tests are code. And code has bugs. Very often the test won't actually be testing what you think its testing. So how do you test a test? A very simple way is to apply the test and make sure it fails, apply your code patch, then re-run the test to make sure it passes. This can all be automated and should reduce the number of tests which don't work. Process ------- A simple patching wrapper can be written (or extend the existing wrapper around patch that perl5 uses) which automates this process. 1) Apply the test. 2) Run the test. Make sure it fails. 3) Apply the code patch. 4) Run the test again, make sure it passes. Simple. Stupid. And there's no reason this can't be applied to Perl5. Problems -------- It is necessary to enforce one patch, one test in order to make this technique feasible. So this RFC is linked to my other RFC about A Test For Every Patch. Things to do NOW ---------------- - Write the testing patching program and/or extend perl5's existing patch wrapper, whichever is easier. [Up for grabs] - Begin using this technique on perl5 (independently, or with p5p's blessing) and gather statistics on how often tests fail to test what they're testing. [Up for grabs, too] Blame ----- All blame for this RFC can be transfered to Michael G Schwern <[email protected]>. -- Michael G Schwern http://www.pobox.com/~schwern/ [email protected] Just Another Stupid Consultant Perl6 Kwalitee Ashuranse But why? It's such a well designed cesspool of C++ code. Why wouldn't you want to hack mozilla? -- Ziggy