Re: Re: Lispworks and Oracle

Kevin Rosenberg <[email protected]> writes:

> > 2. Procuring installations of the various supported lisp environments
> >    and RDBMS systems with build engines, or at least somewhat frequent
> >    HEAD builds by maintainers of those ports.  This way non-portable
> >    changes will be quickly identified.
> 
> I have help with that. I have licenses for ACL and LW on Linux i386
> and Windows. Also, I run SBCL on powerpc and sparc linux.

I'll workup a grid of Platform/OS/RDBMS later so we can better track
our coverage.

> > If this plan is acceptable to people I will start by laying out the
> > points the regression suite needs to cover and start writing the bits
> > I can.  I will also need volounteers outside of onShore Development
> > who would be willing to have an instance of the USQL regression tests
> > running against the HEAD on a platofrm other than linux CMUCL.
> 
> It's acceptable and a good idea. Sure, I'll be glad to help.

Here is the first draft of a document describin the test-suite and the
first orderof overage needed:


* Regression Test Suite Goals

The intent of this test suite is to provide sufficient coverage for
the system to support the following:

** Refactoring and Redesign of particular subsystems

Refactoring and redesign efforts are normally restricted to a single
subsystem, or perhaps to interdependent subsystems.  In such cases, a
set of regression tests which excercise the existing interface of the
rest of USQL to the changing subsystems should be in place and passing
before the coding starts.

** Ensuring portability and Supporting new ports.

The more coverage the test suite provides the easier portability is to
maintain, particularly if we have instances of the test suite running
against the head on the supporting lisp environment/OS/hardware/DBMS
combinations.  Since no individual within the project has the ability
to run all of those combinations themselves, we aredependent upon some
informal coordination between the mintainers of the various ports.
There is also the possibility of using Tinderbox as a build monitor
and allowing others to set up remote builders.

** Adding new RDBMS backends

The entire USQL DBMS interface needs to be excercised by the test
suite, such that a new RDBMS backend that passes all the tests canbe
reasonably assured of working with the USQL layers above that.  These
tests should also serve as impromptu documentation for the details of
that interface and what it expects frothe RDBMS driver layers.

** Bug identification and QA

As new bugs are identified, they should have a regression test written
which excercises them. This is to ensue that we donot start
backtracking. These tests by theselves are also very valuable for
developers, so even if you cannot fix a bug yourself, providing a
testto excercise it greatly reduces the amount of timea developer must
spend finding the bug prior to fixing it.

* Test design issues

** Multiple RDBMS Issues

USQL supports several RDBMS backends, and it should be possible to run
every test against all of them.  However, there are some features
which we want tests for but which are not implemented on several of
the backends.  In those cases, the with-test-db macro allows you to
specify which backends are supported, and it will throw a warning, yet
pass for non-supported backends.  For example, to define a test which
is only supported on postgresql presently:

(defregression ('sequences 1)
  "create a sequence"
  (with-test-db (:postgresql)
    (let ((seqname (gensym))
          (val1 nil))
          
      (create-sequence seqname :schema "sql-tests")
      (setf val1 (sequence-next seqname))
      (unwind-protect
           (unless (< val1 (sequence-next seqname))
             (error "Sequence did not increment properly."))
        (drop-sequence seqname)))))

This test passes on ALL backends, however, it will throw a warning on
non postgresql backends:

    Warning:  This feature is not supported on ORACLE databases


** Test Hygiene

Tests should be able to be run multiple times against the same
database.  It is also important that they clean up after themselves
when they create tables, sequences or other pesistent entities in the
RDBMS backends, because often there are limits to the number of those
thatcan exist at one time, and it also makes debuging thru the SQL
monitors difficult when there aretons of unused tables lying around.

Tests dealing with the object interface can use the existing
"sql-tests" schema when defining view classes they willbe
manipulating.  The view classes will stick around between tests, as
will their contents.  You can use DROP-SCHEMA to wipe out the existing
schema and reloadthe table definitions.  Indexes and sequences can
also be associated with a schema when created, they will be remade as
well (note that sequences will be reset in such cases).

If test need to load large datasets, they should have a mechanism to
ensure the dataset is loaded just once, and not with every test run.

Lastly, because there are various idiosyncracies with RDBMSs,
pleaseensure that you run the entire test suite once when you write
your tests, to ensure that your test does not leave some state behind
which causes other tests to fail.


** Test Run Configuration

The file test-utils.lisp defines several variables which can be used
to control the connection dictionary of the database against which
tests will be run.  New DBMS backends will need to modify the
sql-tests-connection-dictionary so that it builds the correct
connection dictionary.

* Database Connections/Lifecycle

** CreateDB
   *** Without existing DB
   *** With existing DB and use old
   *** With existing DB and use new
   *** Error if existing DB

** Data Definition
  *** Create Tables/Sequences/Indexes -- Should cover creation of
      tables with all supported types of fields.
  *** Delete Tables/Sequences/Indexes
  *** Inspection of Tables and attributes, including types

** Data Manipulation
  *** Update
  *** Insert
  *** Delete
  *** Query

** Functional Interface
  *** Creation/Modification of SQL expressions
  *** Querying

** Embedded SQL syntax
  *** Excercise all sql operators
  
** Object Interface
  *** View class definition
  *** Object creation/manipulation/deletion
  *** Inter-object Relations

** Editing Contexts
  *** Object Create/Modification/Deletion in a context -- partly covered already
  *** Interaction of multiple contexts
  *** Schema manipulation within a context
  *** Rollback and error handling within a context


-- 
Sincerely,
Craig Brozefsky <[email protected]>
Free Scheme/Lisp Software  http://www.red-bean.com/~craig