Re: Test suite

On Fri, 2005-05-06 at 22:28 +0200, Geert Stappers wrote:
> On Fri, May 06, 2005 at 02:07:55PM -0400, David Malcolm wrote:
> > On Fri, 2005-05-06 at 10:03 +0200, Geert Stappers wrote:
> 
>  [ fixed crasher file back into CVS ]
> 
> > > > Are there any other tests for old crashers we should add into CVS?
> > > > 
> > > > Should we ship these files in the tarball?
> > > 
> > > My first thought: no.
> > > 
> > > On second thought:
> > > 
> > > It can be done. What is needed is an explanation why we ship it.
> > > Currently is this the content of examples/crash/README
> > > 
> > >  Files that cause conglomerate to crash.
> > 
> > Maybe should read:
> > "This is a regression test suite for Conglomerate.  The files here
> > trigger bugs in Conglomerate.  The aim is that Conglomerate should be
> > able to load all of these files successfully."
> > 
> --- README      15 Sep 2003 14:16:11 -0000      1.1
> +++ README      6 May 2005 20:05:08 -0000
> @@ -1,4 +1,7 @@
> 
> - Files that cause conglomerate to crash.
> + This is a regression test suite for Conglomerate.
> + The files here trigger bugs in Conglomerate.
> + The aim is that Conglomerate should be able
> + to load all of these files successfully.
> 
>   It is here for easy reproduction and should _not_ go into distribution.

Thanks for doing this.

I've added a command-line option "--unit-test" to Conglomerate.  When
set it causes Conglomerate to quit (with exit value 0) immediately after
loading any files specified on the command line.  

Attached is a proposed patch to your unit test script which uses the
command line flag and removes the need for the 30 second delay.

Though I think I'd prefer it to be silent if possible for success and
only warn about files where there were problems - this probably means we
need to make Conglomerate rather more quiet altogether though.

What do you think of the patch?

> 
> > > 
> > >  It is here for easy reproduction and should _not_ go into distribution.
> > 
> > I think the easiest way to do that is to keep them in CVS but not have
> > them in the tarball.
> 
> Done
> (a long time ago :-p )
> 
> > > > Anyone got thoughts on automating the process?  A shell script that
> > > > looped through all the test files and launched Conglomerate on each in
> > > > turn and somehow reported if Conglomerate crashed on any would be cool!
> > > 
> > > I made this
> > 
> > This looks great!  Thanks!  Can we get the script in CVS please?
> 
> Euh, it allready is ...
> 
> > Ideally we'd have this set up so it gets run if you do a "make test"; it
> > would run the built conglomerate executable upon all of the test cases.
> 
> mmmm, that is a tricky request
> 
> examples/crash is not in the tarball,
> so the toplevel Makefile will complain about it when it is missing.
> 
> request is IMHO too hard to honour
> 
> > 
> > > <screenshot>
>  [ tester.batch ]
> > > </screenshot>
> > > 
> > > It produces this output
> > > 
> > > <screenshot>
> > > stappers@bahrain:/usr/src/conglomerate/examples/crash
> > > $ ./tester.batch
> > > ./tester.batch: line 32:  3803 Terminated  conglomerate $f >$f.log 2>&1
> > > ./tester.batch: line 32:  3809 Terminated  conglomerate $f >$f.log 2>&1
> > > ./tester.batch: line 33:  3815 Terminated  conglomerate $f >$f.log 2>&1
> Those are avoided in the new version.
> 
> > > ****************
> > > rivet.xml.log
> > > ****************
> > > ** Message: Found display spec based on matching public ID of DTD:
> > >     -//OASIS//DTD DocBook XML V4.1.2//EN
> > > 
> > > ** Message: Trying to load
> > > "/usr/share/pixmaps/conglomerate/cong-note-48.png"
> > > ** Message: Trying to load
> > > "/usr/share/pixmaps/conglomerate/cong-note-48.png"
>     ** Message: file now completely load
> > > ****************
> > > septentrio.xml.log
> > > ****************
> > > ** Message: coverage of Website Layout = 0.000000
> > > ** Message: coverage of TEI Lite = 0.500000
> > > ** Message: coverage of Webpage = 0.500000
> > > ** Message: coverage of XHTML 1.0 (strict) = 0.000000
> > > ** Message: coverage of XSL stylesheet = 0.000000
>     ** Message: file now completely load
> > > ****************
> > > testcase-150368.xml.log
> > > ****************
> > >     -//OASIS//DTD DocBook XML V4.1.2//EN
> > > 
> > > 
> > > ** ERROR **: file cong-document-traversal.c: line 594
> > > (recursive_create_traversal_nodes): assertion failed:
> > > (should_have_traversal_node (xml_node))
> > > aborting...
> > > some real report
> > > stappers@bahrain:/usr/src/conglomerate/examples/crash
> > > </screenshot>
> > 
> 
> So what I like to see, is the addtion of this line of code
>     g_message ("file now completely load\n");
> at the right place.
> 
> 
> Cheers
> Geert Stappers
> 
> _______________________________________________
> Conglomerate-devel mailing list
> [email protected]
> http://lists.copyleft.no/mailman/listinfo/conglomerate-devel

_______________________________________________
Conglomerate-devel mailing list
[email protected]
http://lists.copyleft.no/mailman/listinfo/conglomerate-devel

Index: tester.batch
===================================================================
RCS file: /cvs/gnome/conglomerate/examples/crash/tester.batch,v
retrieving revision 1.2
diff -u -p -r1.2 tester.batch
--- tester.batch	6 May 2005 20:24:48 -0000	1.2
+++ tester.batch	17 May 2005 04:27:05 -0000
@@ -8,16 +8,10 @@ TO_TEST=$( ls *.xml )
 runner () {
 for f in ${TO_TEST}
 do
-	conglomerate $f > $f.log 2>&1 &
-	sleep 30  # allow the application to run ( or to chrash ;-)
-	terminate $f  # quit the application
+	conglomerate $f --unit-test > $f.log 2>&1
 done
 }
 
-terminate () {
-kill $(ps -ef | grep $1 | awk '/conglomerate/ { print $2 }')
-}
-
 analyzer () {
 for f in ${TO_TEST}
 do
@@ -29,7 +23,7 @@ done
 }
 
 # main
-runner 2> runner.log # will contain "Terminated" messages
+runner 2> runner.log
 analyzer
 
 echo some real report