Re: unit tests

Kirk Bauer wrote:
> Use the log repository and if the output doesn't match the expected
> results then send an email which they could then confirm and update
> the expected results.  My original problem was that for each new log
> entry the developer would have to update the script and the expected
> results which seems like a bit of work to do for each release.

I see what you are saying. I don't think running the tests against a big 
pool of unpredictable log files is the way to go. It'll result in the 
problems you've stated.

Each unit test should have its own, private, carefully controlled set of 
data files. If down the road that data is determined to be inadequate to 
cover some case, then you add a regression test or otherwise add new 
data to the test data and a corresponding test change. This all happens 
under the direction of the maintainer of that script.

What value do you see in running tests against a pool of pool of 
unpredictable log files? I could see it being useful for a simple system 
test to make sure nothing blows up. In that case your test code would 
only be looking for code that dies, returns error codes, or triggers a 
SIGPIPE.

As before, the biggest value I see in a repository of log files is that 
it provides a convenient place for developers to extract relevant log 
file samples to be turned into static data files in their tests. It may 
ease the process of letting end-users contribute log files. It could 
track meta data on the logs, like OS name, version, application name, 
version, etc., and assist in making the collection more diverse by 
declining samples for which there already is 10,000 lines (or whatever) 
in the repository.


> I was just hoping to take some burden off of the developers.

Only the developer of the script or someone who has the equivalent 
knowledge is in a position to determine what the correct output is for a 
given input, so they should be the ones controlling what data gets fed 
into a test and determining what output to verify.


> I may be up for writing some of this...  If somebody else is
> interested then I'll support you, if not hopefully I can get around to 
> it soon.

I'm up for writing tests for the pieces of code I touch, but as I've 
noted in the past few emails, the first step is coming to agreement on a 
few standards. The tests themselves can be written in small increments.

If no one has an opinion on how to structure the tests, I'll expand my 
prior suggestions into something more specific and post that for comment.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/