Re: unit tests

Kirk Bauer wrote:
> The normal method of running the code against golden data and
> comparing the results to expected results only works on the actual
> log entries in the golden logs.

I'm not sure I see the problem. As end-users report incompatibilities 
and submit corresponding log lines, you would treat it like any other 
regression test and add it to the collection of test data for that script.


> Effectively every time a log entry is added or changed or every
> time the script is changed then the expected result would have to be
> changed.

If the report format changes, it is a given the unit tests will have to 
change.

If additional data files are added, then it is a given that an 
additional test will need to be written to compare the produced report 
against.

This all seems like pretty typical testing procedures, no?


> An idea I have been thinking about is a log entry repository with a
> web interface.  This would allow developers to view example log
> entries and users to submit unmatched entries online.
> 
> Then before a release the entire log repository would be run
> separately against each service script at predefined detail levels and
> the reports mailed to the script's developer(s) who would have to read
> through the report and make sure it is correct and doesn't have any
> unmatched entries, etc.

The repository of logs is not a bad idea, but doesn't seem instrumental 
to pulling off automated testing. It would be of more value to 
developers writing new code and needing a handy source for test data 
when creating their unit tests.

Having reports emailed to developers for manual verification seems like 
a bad idea, especially if automated tests can do the job adequately.

I did have the thought that we'd need some infrastructure for generating 
log data that has current time stamps if we want to test the date range 
selection code, but that requirement only applies to the main logwatch 
script (when being tested as a system) and the various date filter 
scripts. The various service filter scripts don't care about dates, as 
far as I've seen, and could use static data.

I'd recommend starting by writing tests for the lowest layer components 
that have the most impact, which is probably the shared filters. Then 
move on to the service filters, and logfile filters. Finally unit and 
system tests for the main script.

But as I mentioned previously, figuring out the details of how the tests 
will be implemented, ran, and located, is necessary before anyone can 
start writing tests.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/