test suite driver enhancements

 > 1) A couple of tests should be skipped (or with an acceptable failure) on
 > specific platforms. Example:

The https test evidently should be skipped on all platforms so far.

It would be nice to at least collect a list of the tests skipped so
at the end you can see which ones were skipped, perhaps report the 
number skipped along with the number passed and failed.

 > 2) When a group of tests belongs together, and a test fails, skip
 >    the remaining tests of the same group.

So when one fails you set some variable that causes others to be skipped.
You could, of course, consider the entire group to be a single test.
And then if it fails, have a way to view it as a sequence of separate
tests.

 > 3) When a test is skipped, we need to write the condition twice:
 >    #+condition form #+condition result

First, #+ syntax is bad for collecting a list of skipped tests.
Second, I don't yet see why you dislike deftest forms.
I'd expect a deftest form to contain a condition as one of its arguments.

 >  This means, minimal clutter. Ideally, no need to write 'deftest' forms,
 >  'assert' statements etc.

(deftest condition form result)
seems less clutter than 
#+condition form #+condition result

 >   This means, use SETQ/DEFVAR, not LET/LET*, to create temporary objects
 >   that the rest of the test references.

This seems a fairly minor consideration.
We face the same problems for debugging normal code all the time.

 > What I do not want is a test suite style like these:
...
 > In other words, I see the test suite files as *data* (to be
 > interpreted by the test suite driver), not as a Lisp program.

I don't understand why you dislike the sample forms or why they are
not data to be interpreted just as much as what you propose below.

 > To solve problem 1), I propose to add - in tests.lisp - OS names to the
 > *features* list. So that it becomes possible to write #+(or AIX HP-UX).
 > Outside of the test suite, i.e. for user programs, there should be no
 > distinction between Linux, AIX, HP-UX, etc.

(See complaints about #+ above.)
If you avoid #+ then it doesn't really matter whether you store such
data in *features* or some other global variables.

 > To solve problem 2), I propose a syntax with braces:
 > { FORM1 EXPECTED-RESULT1 FORM2 EXPECTED-RESULT2 ... }

Why do you want to leave lisp syntax?  This only makes it harder for
someone to use the test source file.  One should be able to easily 
write code that just reads all the forms from the file without having 
to deal with new syntax.
I don't see how this can be better than something like
(as-one-test FORM1 EXPECTED-RESULT1 FORM2 EXPECTED-RESULT2 ... )
And better than that, I think, would be
(as-one-test (deftest ...)(deftest ...) ...)

 > To solve problem 3), I propose a syntax like this:
 > $+CONDITION FORM EXPECTED-RESULT

Again it seems that leaving lisp syntax is only making trouble
I don't see how this is any better than
(deftest CONDITION FORM EXPECTED-RESULT)


It seems to me that you can view deftest and as-one-test forms
either as lisp programs (if you define them as macros and then load
the file) or as data if you write a program that reads all the forms
in the file and then interprets them in some way other than simple
eval.  Defining the macros seems to me the easiest way of just running 
the tests, but (1) there are other things you might do, like just 
count the tests, and (2) even for running the tests you probably want
to do things like reporting the number of passes, failures, skips.
In fact different uses of the same test suite might be conveniently
implemented by using different definitions of those same macros.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
clisp-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/clisp-devel