Re: unit tests

Kirk Bauer wrote:
> I was envisioning the ability for end users to submit unmatched
> entries via the web. 

This is a good idea. But it involves some infrastructure that someone 
will need to set up, and I don't see it as a necessary precondition to 
getting started with unit tests.


> These log entries would instantly become part of the unit testing. 

I made a case against that in the previous message:

>> Each unit test should have its own, private, carefully controlled set of
>> data files. If down the road that data is determined to be inadequate to
>> cover some case, then you add a regression test or otherwise add new
>> data to the test data and a corresponding test change. This all happens
>> under the direction of the maintainer of that script.
...
>> Only the developer of the script or someone who has the equivalent
>> knowledge is in a position to determine what the correct output is for a
>> given input, so they should be the ones controlling what data gets fed
>> into a test and determining what output to verify.



> Before a release the developer either needs to approve the new output
> (presumably without unmatched entries) or invalidate the log entry 
> that the user submitted. 

I understand what you are trying to accomplish, but I think it would be 
better handled (and easier to implement) with a more controlled approach 
to getting new log samples added to unit tests.

Really the goal you are striving for is expediting the flow of feedback 
(reports of unmatched entries) from end-users to developers of the 
relevant service script. It's a good idea, but we could easily become 
mired in the task of setting that up. The volume seems to be low enough 
right now that it should be adequate simply having a developer list to 
receive reports of unmatched entries, and letting developers "pluck" the 
desired samples from that list for inclusion in their unit tests.


> Over time we will have a big log repository, automated unit testing,
> and a better feedback system for users...

Right. But each of those is a potentially big project. I say we start 
simply with the automated testing first.


I wrote:
>> If no one has an opinion on how to structure the tests, I'll expand my
>> prior suggestions into something more specific and post that for comment.

No opinions have been forthcoming, so I'll write something up.

Also, I'd like to request commit access to CVS for the purpose of adding 
unit testing and committing my rewritten dovecot service filter.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/