Re: Suggestion for ASDF:TEST-OP

Gary King wrote:
> Hi Robert,
> 
> I heartily endorse the spirit of this proposal. I would, however,  
> like to add to it.
> 
> In order for automated testing to function as part of a build/make  
> system, there must be some way for the system to know the results of  
> the testing. Did all tests pass? Were there failures? Were there  
> errors? Was every test that I care about actually run (this last  
> makes sense in the case where the tests run can be configured; it  
> isn't much use to automate testing if you can skip all the important  
> tests by accidently using the wrong configuration file!).
> 
> Therefore I think that the following is an additional desideratum for  
> the ideal 'test-op
> 
> 1. (asdf:oos 'asdf:test-op <system>) should return a single whose  
> meaning is as follows:
> 
>      * t - all passed
>      * nil - something unspecified went wrong
>      * :errors - there was at least one error during testing
>      * :failures - there was at least one test failure

The reason I didn't propose this is that it would require a radical
restructuring of ASDF to make any of the operations return anything of
interest.

So in the interest of getting a short-term improvement I proposed only
the simpler modification --- admitting a stream for a report --- that
would at least give something a human could eyeball.  I propose we do
that reasonably quickly, and then move to a longer-term solution of
getting return values when we start fixing asdf:traverse...

A minor nit about your return value:  I think either there should be
*two* values, or the success should return NIL.  If t is all passed,
then the conditional is awkward....

Also, :errors and :failures are not mutex, so probably something more like

(values success errors failures)

is the right return...

> 
> 2. If a test system is able to, it should return more information  
> about exactly which tests were run and their individual stati in the  
> second return value. This is optional behavior.
> 
> 
> On Nov 12, 2007, at 11:00 AM, Robert Goldman wrote:
> 
>> Here's a tentative proposal for extension:
>>
>> Add to the test-op a test-result-stream slot and initarg (this is a
>> *tentative* proposal --- if it were to be approved, someone should  
>> come
>> up with something less verbose) that would default to t.
> 
> Actually, I like test-result-stream
> 
>> Explicitly declare that the test results should be written to this  
>> stream.
> 
> Yes.

What I should have said is that we should add this to a docstring for
test-op and to the ASDF manual.

Does that sound OK?


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/