Re: Common DBI Driver Test Suite - Requirements

On Fri, Sep 27, 2013 at 11:42:10AM +0200, Jens Rehsack wrote:
> Am 27.09.2013 um 00:03 schrieb Tim Bunce <[email protected]>:
> 
> > On Wed, Sep 25, 2013 at 08:36:22PM +0200, Jens Rehsack wrote:
> > 
> >> For me, a G10 is interesting (handling SQL::Statement as a DBD)
> >> 
> >> G10: Provide tools/infrastructure to combines several test cases in
> >>     particular order into tests to allow workflows being tested
> >>     and stress tests by looping can be performed.
> > 
> > Also here. I'm not quite sure what problem this is trying to address.
> 
> 
> This is more or less an idea I had during test development. For a lot
> of tests, we write some sequences of statements to finally prove one
> or a few things working fine (at least for DBD::CSV, DBD::DBM and a
> lot of SQL::Statement tests - but IIRC, this is true for SQLite, too).
> 
> So for testing whether UPDATE/DELETE works, we create some tables, do
> some inserts and after that, modify them. For SELECT testing, same.
> For ChopBlank testing (via select, of course), similar. For JOIN testing,
> more tables - but finally the same. This goes on … (different edge cases
> to test, but finally all the same).
> 
> For SQL::Statement I added a performance check )for being run through
> the profilers we have) which is assembled from one or more of the above
> tests. But by hand.
> 
> If there is a design of combining test cases to a sequence of tests (eg.
> create tables -> fill data -> select -> update -> delete -> drop), it
> could help simplifying test cases and allow put loops around sequences
> to redo some test sequence for 100 times for profiling.
> 
> The design finally choosen for this goal might be a composite for test
> cases - but the test case design shall allow doing that.

Test files need to be self-contained, so they can be run in parallel and
in randomized order.

They'd naturally have a setup-fixture phase, a run-tests phase, and a
teardown-fixture phase.

Repeating the whole sequence of phases N-times would be easy enough.
For some, but not all tests, repeating the run-tests phase N-times
between the setup and teardown phases would be possible, and faster.

I think this ability would fall-out nicely from defining a suitable API
to the code in the test modules. Then it's a simple change to the code
that calls the subs/methods in the test modules.

Tim.

p.s. I'll add parallel running to the requirements - it has significant
implications on the design, e.g., unique names for tables.