Re: Return of [a] Common Lisp Test Suite

> * In message <[email protected]>
> * On the subject of "Re: Return of [a] Common Lisp Test Suite"
> * Sent on Thu, 17 Oct 2002 05:44:19 -0500
> * Honorable "Paul F. Dietz" <[email protected]> writes:
>
> Peter Van Eynde wrote:
> 
> > A bit of history: the clocc ansi-test is sort-of based on the clisp test
> > together with a few tests I wrote and test-cases based on reported
> > clisp, cmucl and sbcl bugs.
> > I try to keep it up-to-date with the clisp CVS, but I'm way behind :-(.
> > I have no objections at all to you getting write access, in fact the
> > more committers the merrier :-).
> 
> My one concern is what is the policy for resolving differences of what
> constitutes standard CL.  Is there a mailing list for this?  A set of
> [W|Tw|Cl]iki pages?

the test itself :-)
seriously, the each test in the CLOCC ansi-test has an explanation
section which should explain what the test checks and why the expected
results are as they are.

> Also, I noticed the clocc ansi-test suite has some conditionalization
> on implementation.  Why is that?  Presumably if it is testing the
> standard, the standard is the same regardless of the implementation.
> I refer in particular to test :characters-legacy-889, which expects
> different results in different implementations (IMO, the CMUCL
> expected result is contrary to the standard):
> 
> (check-for-bug :characters-legacy-889
>    (char-name #\Newline)
>    #-cmu
>    "Newline"
>    #+cmu
>    "Linefeed")

CLOCC ansi-test come from the CLISP tests (which was based on some other
tests).  the CLISP test suite is both ANSI compliance test suite and a
regression test.
In many cases the result is left to the implementation, so
conditionalization is reasonable.

Note that just saying "we are a compliance test, not a regression test"
and dropping the conditionalization is not a good idea.

In fact, I would suggest that for each supported implementation you
keep a list of "known failures" so that it would be easier to use your
test for regression testing.

Note that this is different from conditionalization: if an
implementation is free to return 1.5430807 and 1.5430806 as the result
of (COSH 1), it is reasonable to conditionalize.  But no implementation
is free to return 4 as a result of (1+ 2), so the result of (1+ 2)
should not be conditionalized, but if there is an implementation which is
known to return an incorrect value, this test should be added to the
list of "known failures" for this implementation, so that the test
results can be reported not as "failed 60 tests out of 4123: ..." but as
"out of 40 known failures, 38 (...) still fail but 2 (...) have
apparently been fixed and now pass, but there are 4 (...) unexpected
failures (total 42 tests failed out of 4123)" ["(...)" means the list
of the tests that fail/pass].

Also, CLOCC includes "clunit" - a package for unit testing, and it
might be a good idea to convert the unified CLOCC ansi-test suite to
use it.

I urge you to move this discussion to <clocc-devel>.
After the other 2 CLOCC admins agree (this won't be a problem), you
will be added as a developer to CLOCC. what is your SF handle?

-- 
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
An elephant is a mouse with an operating system.