Re: unit tests

I love this discussion so far but have been on the road and in fact
this email is from my BlackBerry so I will be more brief than would
normally be appropriate for the number of good points brought up here.

I have been fairly uninvolved with Logwatch for years now.  Pretty
much Mike T and MrC do some development of their own as well as
applying the patches sent in by others.  Then they tell me when and I
do a build.  Unfortunately this makes me slack off on release notes.

I would like to see some improvements.  At one point I wanted to get a
volunteer for each service who could maintain that script.
Unfortunately the idea didn't really take off.  At least for some
scripts one person pretty much maintains it.

Validating the filters is tough in general.  The normal method of
running the code against golden data and comparing the results to
expected results only works on the actual log entries in the golden
logs.  Effectively every time a log entry is added or changed or every
time the script is changed then the expected result would have to be
changed.

An idea I have been thinking about is a log entry repository with a
web interface.  This would allow developers to view example log
entries and users to submit unmatched entries online.  It would ask
users for information such as OS version and version of the app that
generated it.  Developers can then get a list of all log entries for a
service or just for certain versions of the program, etc.  There could
even be a nice command-line tool for accessing this repository.

Then before a release the entire log repository would be run
separately against each service script at predefined detail levels and
the reports mailed to the script's developer(s) who would have to read
through the report and make sure it is correct and doesn't have any
unmatched entries, etc.

That is my idea, thought I would share.


On 6/28/07, Tom Metro <[email protected]> wrote:
> MrC wrote:
> > Tom Metro wrote:
> >> (BTW, any thoughts on writing unit tests for these scripts?)
> >
> > I too asked about this some time ago.  The general feedback was
> > negative; so I wrote my own basic test data in the amavis and postfix
> > scripts.
> >
> > I would personally like to see  (in no particular order):
> >
> > 1) test logs and golden files
>
> Sample data can always be improved upon later. It seems the first step
> is agreeing how to structure the tests and where to place them in the
> project directory.
>
> Maybe have a tests/ top-level directory (parallel to the scripts/, lib/,
> project/, etc. directories). Each script could have it's own directory
> to contain its tests and test data, so for example, the Dovecot service
> filter:
> tests/scripts/services/dovecot/{data.log,test.t}
>
> Then we'd need to agree on a test runner (like prove[1]) and perhaps
> create a script or Makefile at the top level to conveniently run the
> unit tests.
>
> 1. http://search.cpan.org/~andya/TAP-Parser-0.51/bin/prove
>
> Next I'd consider whether it makes sense to create some common
> infrastructure for the tests. For example, for almost every service
> filter you'll want to run the tests at a minimum of the three standard
> detail levels. A common infrastructure might iterate over all the *.log
> files in the script's test directory, pipe the data into the script, and
> pass the output to a method named <logfile>_<detailLevel> for validation.
>
> I'd drive this process from the .t file, so you'd be free to skip the
> standard stuff if not applicable, or supplement with additional tests.
> The end result may be just a helper library, like Test::More, that
> provides functions like:
>   pipe($script,$data_file,&validate,"$data_file: valid for detail level
> %d");
>
>   -Tom
>
> --
> Tom Metro
> Venture Logic, Newton, MA, USA
> "Enterprise solutions through open source."
> Professional Profile: http://tmetro.venturelogic.com/
> _______________________________________________
> Logwatch-Devel mailing list
> [email protected]
> http://www2.list.logwatch.org:81/mailman/listinfo/logwatch-devel
>
>


-- 
Kirk Bauer <[email protected]>
http://linux.kaybee.org | www.autorpm.org | www.logwatch.org