Re: Disabled tests - known bugs or not yet implemented features?

OK, I see the problem. A basic rule of HttpUnit through its  
development has been to follow the agile development approach, which  
among other things says: unit tests must always run at 100%. Unit  
tests verify that the code does what the developers believe it does.  
Test should never be committed to the baseline unless they pass, and  
regressions in the unit tests are not permitted. Under certain very  
extraordinary circumstances, we may disable a test - usually this is  
only when that test is found to be intermittent or platform-specific.

There is a second kind of tests for which the rules you are describing  
would make sense - acceptance tests. It makes perfect sense for  
applications to run acceptance tests at less than 100% and track  
increasing success rates as a measure of completeness. As a library,  
HttpUnit has never used those. Instead, the approach has been to use  
the bug database as an indicator of problems not yet solved. That  
means that we do not close bugs simply by creating tests and committed  
them to the repository, commented out. Unless the feature actually  
works, the bug report should remain open.

It would make sense, I think, to adopt an approach of having a second  
suite of new tests that correspond to open bugs. These would not be  
part of the 'ant test' target, but could be added as those bugs are  
fixed. The feature priorities you mention could be tied in nicely with  
this second suite.


On May 15, 2008, at 10:07 AM, Wolfgang Fahl wrote:

> The dicussion went something like:
> RG> I note, however, that we    now have 12 disabled tests. > This  
> makes me very uncomfortable. I would
> RG> like to place a priority on fixing those - ahead of addressing  
> any new
> RG> bug reports, patches, or RFEs; however, as long as we are going  
> to do
> RG> that, I see no reason that we shouldn't release 1.7 ahead of that.
> WF> The priority setting is fine with me - we could even show the  
> priority in the disabled message by adding a parameter.
> RE> By "place a priority" I meant that we should not be working on  
> other things until these are fixed...
> A key question for handling the failing test is what does the  
> failure indicate?
>  - a known bug?
>  - a not yet implemented feature?
>
> To be a known bug there must be a promised feature that doesn't  
> work. E.g. I currently reckon httpunit 1.7 does not promise for full  
> Javascript support. It should be better than 1.6.2 - but that's it  
> then.
>
> For a not yet implemented feature already having a test case is nice  
> - but that it fails only shows where we are from a milestone point  
> of view.
>
> >From my point of view not all of the potentially failing tests  
> above are alike regarding urgency and priority.
>
> Priority is interpreted by me as:
> A: Must have this feature
> B: Should have this feature
> C: A feature that would be nice to have
>
> Urgency is interpreted by me as:
> 1: ASAP
> 2: On next occasion
> 3: When time/resources allow for it
>
> >From my point of view not all of the potentially failing tests  
> above are alike regarding urgency and priority. Here is what my  
> rating is:
>
>
>  The following tests are not active - the features tested are not  
> part of the current release:
>  #  |        testname               | priority | urgency | reason
> ----+-------------------------------+----------+--------- 
> +----------------------------------------
>   1 |                 testIFrameBug |        B |       2 | patch  
> needed for 'No frame named iframe_after_lessthan_in_javascript is  
> currently active'
>   2 |          testParamReplacement |        A |       2 | bug  
> 1393144 pending - waiting for patch
>   3 |                 testCloneNode |        B |       3 | not fixed  
> for old javascript engine
>   4 |  testJavascriptDetectionTrick |        A |       2 | waiting  
> for nekoHtml bug #1932445
>   5 |                       testDOM |        B |       3 | not fixed  
> for old javascript engine
>   6 |        testModifySelectLength |        B |       3 | not fixed  
> for old javascript engine
>   7 |           testLargeJavaScript |        C |       2 | fails  
> with class format error for 10000 lines at optimizationLevel 0 the  
> default is level -1 so we only warn
>   8 |           testLargeJavaScript |        C |       2 | fails  
> with class format error for 10000 lines at optimizationLevel 1 the  
> default is level -1 so we only warn
>   9 |         testArgumentsProperty |        B |       3 | not fixed  
> for old javascript engine
>  10 |             testCreateElement |        B |       3 | not fixed  
> for old scripting engine
>  11 |               testIsCommitted |        B |       2 | bug  
> report 1534234 is pending - waiting for testcase/improved patch
>  12 |                 testFormTable |        A |       2 | for  
> pending bug 1043368
>
>
> Non of the tests got an A1 from me which would mean it must be  
> immediately fixed before the 1.7 release.
>
> Some features are strictly javascript related where we no that the  
> new Scripting engine is key to delivering - so that would be  
> something for a 2.0 release and we'd need developer resources for  
> that - currently nobody seems to be working on this and I personally  
> don't have any need for enhanced Javascript at this time and  
> therefore I am not too motivated to work much on this.
>
> Also this is just my own view on it - I think the httpunit user and  
> developer community should have a say in this.
>
> Yours
>   Wolfgang
>
>
> BITPlan - smart solutions
> Pater-Delp-Str. 1, D-47877 Willich Schiefbahn
> Tel. +49 1805 - BITPLAN / +49 1805 248 752, Fax +49 2154 811-481
> Web: http://www.bitplan.de
> bitplan GmbH, Willich - HRB 6820 Krefeld, VAT-ID: 10258040548,  
> Geschäftsführer: Wolfgang Fahl
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/_______________________________________________
> Httpunit-develop mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/httpunit-develop

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft 
Defy all challenges. Microsoft(R) Visual Studio 2008. 
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Httpunit-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/httpunit-develop