Re: make testing more accessible to contributors

Stefan Seefeld <[email protected]> writes:
> When trying to confine a bug I tried to first see whether it was already
> covered by an existing test. Unfortunately, I found the existing test
> infrastructure a bit confusing / hard to use.

How about "idiosyncratic"?

> Is there anything that could be done to make that easier ? Ideally
> I would only need to have to check out an additional module (not two !)
> and then run 'make check' (or similar) in it.

The test definition and the test results documents conform to the DTD
developed for XSL FO tests for the tests done for the XSL 1.0 Candidate
Recommendation stage.

Multiple vendors and NIST contributed test suites to the XSL 1.0 CR testing.
Multiple vendors each ran multiple test suites and produced documents listing
how well their implementation faired with each test.

The 'testsuite' and 'testing' modules are separate so, in principle, the
'testing' module could be used with any testsuite and the 'testsuite' module
could be used by anyone who is set up to use XSL 1.0 CR testsuites.

The scripts and XSLT stylesheets in the 'testing' package do three things:

 - Generates the shell script that can run all the tests

 - Creates summary and individual HTML reports for the tests

 - Updates results data from the form in each individual test result page

Updating the results is clunky and requires that you regenerate all the HTML
to see the results reflected in the HTML, but it's a lot more convenient than
updating results by hand.

And, yes, it could be improved.

> I did glance over some READMEs both in 'testsuite' as well as 'testing',
> but couldn't quite get it to work.

Try 'make-success-report.sh'.

There isn't a Makefile target because, with the testsuite in a separate
directory and the test FO files never referred to in any way sensible to
'make', there's no way that 'make' could manage the dependencies and reliably
run the tests only when necessary.

Regards,


Tony.



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642