Re: unit tests

Maybe a hybrid system makes sense.  Use the log repository and if the output
doesn't match the expected results then send an email which they could then
confirm and update the expected results.  My original problem was that for
each new log entry the developer would have to update the script and the
expected results which seems like a bit of work to do for each release.  I
was just hoping to take some burden off of the developers.

I may be up for writing some of this -- I agree we need to start with the
low-level items first which should be much easier to automate the validation
of than the service scripts themselves.  If somebody else is interested then
I'll support you, if not hopefully I can get around to it soon.

On 6/29/07, Tom Metro <[email protected]> wrote:
>
> Kirk Bauer wrote:
> > The normal method of running the code against golden data and
> > comparing the results to expected results only works on the actual
> > log entries in the golden logs.
>
> I'm not sure I see the problem. As end-users report incompatibilities
> and submit corresponding log lines, you would treat it like any other
> regression test and add it to the collection of test data for that script.
>
>
> > Effectively every time a log entry is added or changed or every
> > time the script is changed then the expected result would have to be
> > changed.
>
> If the report format changes, it is a given the unit tests will have to
> change.
>
> If additional data files are added, then it is a given that an
> additional test will need to be written to compare the produced report
> against.
>
> This all seems like pretty typical testing procedures, no?
>
>
> > An idea I have been thinking about is a log entry repository with a
> > web interface.  This would allow developers to view example log
> > entries and users to submit unmatched entries online.
> >
> > Then before a release the entire log repository would be run
> > separately against each service script at predefined detail levels and
> > the reports mailed to the script's developer(s) who would have to read
> > through the report and make sure it is correct and doesn't have any
> > unmatched entries, etc.
>
> The repository of logs is not a bad idea, but doesn't seem instrumental
> to pulling off automated testing. It would be of more value to
> developers writing new code and needing a handy source for test data
> when creating their unit tests.
>
> Having reports emailed to developers for manual verification seems like
> a bad idea, especially if automated tests can do the job adequately.
>
> I did have the thought that we'd need some infrastructure for generating
> log data that has current time stamps if we want to test the date range
> selection code, but that requirement only applies to the main logwatch
> script (when being tested as a system) and the various date filter
> scripts. The various service filter scripts don't care about dates, as
> far as I've seen, and could use static data.
>
> I'd recommend starting by writing tests for the lowest layer components
> that have the most impact, which is probably the shared filters. Then
> move on to the service filters, and logfile filters. Finally unit and
> system tests for the main script.
>
> But as I mentioned previously, figuring out the details of how the tests
> will be implemented, ran, and located, is necessary before anyone can
> start writing tests.
>
>   -Tom
>
> --
> Tom Metro
> Venture Logic, Newton, MA, USA
> "Enterprise solutions through open source."
> Professional Profile: http://tmetro.venturelogic.com/
> _______________________________________________
> Logwatch-Devel mailing list
> [email protected]
> http://www2.list.logwatch.org:81/mailman/listinfo/logwatch-devel
>
>


-- 
Kirk Bauer <[email protected]>
http://linux.kaybee.org | www.autorpm.org | www.logwatch.org

_______________________________________________
Logwatch-Devel mailing list
[email protected]
http://www2.list.logwatch.org:81/mailman/listinfo/logwatch-devel