Re: Test suite

On Thu, May 05, 2005 at 01:58:25PM -0400, David Malcolm wrote:
> We've been using the directory: examples/crash to hold test cases.
> 
> I just added a testcase for a bug here and noticed that an old test case
> had been deleted (with a note that the related bug had been fixed, IIRC)
> 
> I think deleting old testcases is a bad idea.  It makes the project much
> stronger if we have lots of simple test documents that we can load, to
> verify that old bugs aren't creeping back to haunt us.  
> 
> Geert: please can we have that old testcase back please

Done

> Are there any other tests for old crashers we should add into CVS?
> 
> Should we ship these files in the tarball?

My first thought: no.

On second thought:

It can be done. What is needed is an explanation why we ship it.
Currently is this the content of examples/crash/README

 Files that cause conglomerate to crash.

 It is here for easy reproduction and should _not_ go into distribution.

> 
> Anyone got thoughts on automating the process?  A shell script that
> looped through all the test files and launched Conglomerate on each in
> turn and somehow reported if Conglomerate crashed on any would be cool!

I made this
<screenshot>
stappers@bahrain:/usr/src/conglomerate/examples/crash
$ cat tester.batch
#!/bin/bash
#
#  script for testing in a batch
#  feel free to modify into another runtime ( e.g. python, expect, perl
#  )
#
TO_TEST=$( ls *.xml )

runner () {
for f in ${TO_TEST}
do
        conglomerate $f > $f.log 2>&1 &
        sleep 30  # allow the application to run ( or to chrash ;-)
        terminate $f  # quit the application
done
}

terminate () {
kill $(ps -ef | grep $1 | awk '/conglomerate/ { print $2 }')
}

analyzer () {
for f in ${TO_TEST}
do
        echo "****************"
        echo $f.log
        echo "****************"
        tail -n 5 $f.log
done
}

# main
runner
analyzer

echo some real report

# end of batch file
</screenshot>

It produces this output

<screenshot>
stappers@bahrain:/usr/src/conglomerate/examples/crash
$ ./tester.batch
./tester.batch: line 32:  3803 Terminated  conglomerate $f >$f.log 2>&1
./tester.batch: line 32:  3809 Terminated  conglomerate $f >$f.log 2>&1
./tester.batch: line 33:  3815 Terminated  conglomerate $f >$f.log 2>&1
****************
rivet.xml.log
****************
** Message: Found display spec based on matching public ID of DTD:
    -//OASIS//DTD DocBook XML V4.1.2//EN

** Message: Trying to load
"/usr/share/pixmaps/conglomerate/cong-note-48.png"
** Message: Trying to load
"/usr/share/pixmaps/conglomerate/cong-note-48.png"
****************
septentrio.xml.log
****************
** Message: coverage of Website Layout = 0.000000
** Message: coverage of TEI Lite = 0.500000
** Message: coverage of Webpage = 0.500000
** Message: coverage of XHTML 1.0 (strict) = 0.000000
** Message: coverage of XSL stylesheet = 0.000000
****************
testcase-150368.xml.log
****************
    -//OASIS//DTD DocBook XML V4.1.2//EN


** ERROR **: file cong-document-traversal.c: line 594
(recursive_create_traversal_nodes): assertion failed:
(should_have_traversal_node (xml_node))
aborting...
some real report
stappers@bahrain:/usr/src/conglomerate/examples/crash
</screenshot>


> 
> Dave


Cheers
Geert Stappers