Re: Test suite

On Fri, 2005-05-06 at 10:03 +0200, Geert Stappers wrote:
> On Thu, May 05, 2005 at 01:58:25PM -0400, David Malcolm wrote:
> > We've been using the directory: examples/crash to hold test cases.
> > 
> > I just added a testcase for a bug here and noticed that an old test case
> > had been deleted (with a note that the related bug had been fixed, IIRC)
> > 
> > I think deleting old testcases is a bad idea.  It makes the project much
> > stronger if we have lots of simple test documents that we can load, to
> > verify that old bugs aren't creeping back to haunt us.  
> > 
> > Geert: please can we have that old testcase back please
> 
> Done

Thanks

> 
> > Are there any other tests for old crashers we should add into CVS?
> > 
> > Should we ship these files in the tarball?
> 
> My first thought: no.
> 
> On second thought:
> 
> It can be done. What is needed is an explanation why we ship it.
> Currently is this the content of examples/crash/README
> 
>  Files that cause conglomerate to crash.

Maybe should read:
"This is a regression test suite for Conglomerate.  The files here
trigger bugs in Conglomerate.  The aim is that Conglomerate should be
able to load all of these files successfully."

> 
>  It is here for easy reproduction and should _not_ go into distribution.

I think the easiest way to do that is to keep them in CVS but not have
them in the tarball.

> 
> > 
> > Anyone got thoughts on automating the process?  A shell script that
> > looped through all the test files and launched Conglomerate on each in
> > turn and somehow reported if Conglomerate crashed on any would be cool!
> 
> I made this

This looks great!  Thanks!  Can we get the script in CVS please?

Ideally we'd have this set up so it gets run if you do a "make test"; it
would run the built conglomerate executable upon all of the test cases.

> <screenshot>
> stappers@bahrain:/usr/src/conglomerate/examples/crash
> $ cat tester.batch
> #!/bin/bash
> #
> #  script for testing in a batch
> #  feel free to modify into another runtime ( e.g. python, expect, perl
> #  )
> #
> TO_TEST=$( ls *.xml )
> 
> runner () {
> for f in ${TO_TEST}
> do
>         conglomerate $f > $f.log 2>&1 &
>         sleep 30  # allow the application to run ( or to chrash ;-)
>         terminate $f  # quit the application
> done
> }
> 
> terminate () {
> kill $(ps -ef | grep $1 | awk '/conglomerate/ { print $2 }')
> }
> 
> analyzer () {
> for f in ${TO_TEST}
> do
>         echo "****************"
>         echo $f.log
>         echo "****************"
>         tail -n 5 $f.log
> done
> }
> 
> # main
> runner
> analyzer
> 
> echo some real report
> 
> # end of batch file
> </screenshot>
> 
> It produces this output
> 
> <screenshot>
> stappers@bahrain:/usr/src/conglomerate/examples/crash
> $ ./tester.batch
> ./tester.batch: line 32:  3803 Terminated  conglomerate $f >$f.log 2>&1
> ./tester.batch: line 32:  3809 Terminated  conglomerate $f >$f.log 2>&1
> ./tester.batch: line 33:  3815 Terminated  conglomerate $f >$f.log 2>&1
> ****************
> rivet.xml.log
> ****************
> ** Message: Found display spec based on matching public ID of DTD:
>     -//OASIS//DTD DocBook XML V4.1.2//EN
> 
> ** Message: Trying to load
> "/usr/share/pixmaps/conglomerate/cong-note-48.png"
> ** Message: Trying to load
> "/usr/share/pixmaps/conglomerate/cong-note-48.png"
> ****************
> septentrio.xml.log
> ****************
> ** Message: coverage of Website Layout = 0.000000
> ** Message: coverage of TEI Lite = 0.500000
> ** Message: coverage of Webpage = 0.500000
> ** Message: coverage of XHTML 1.0 (strict) = 0.000000
> ** Message: coverage of XSL stylesheet = 0.000000
> ****************
> testcase-150368.xml.log
> ****************
>     -//OASIS//DTD DocBook XML V4.1.2//EN
> 
> 
> ** ERROR **: file cong-document-traversal.c: line 594
> (recursive_create_traversal_nodes): assertion failed:
> (should_have_traversal_node (xml_node))
> aborting...
> some real report
> stappers@bahrain:/usr/src/conglomerate/examples/crash
> </screenshot>
>