Re: Proposal: Test::MakeVariantTestFiles

Am 30.09.2013 um 20:03 schrieb Tim Bunce <[email protected]>:

> On Mon, Sep 30, 2013 at 08:52:04AM +0200, Jens Rehsack wrote:
>> 
>> Am 29.09.2013 um 21:26 schrieb Tim Bunce <[email protected]>:
>> 
>>> The DBI adopted an approach towards testing 'variants' by creating extra
>>> test files that act as wrappers around existing test files.
>>> 
>>> The wrappers modify the process in some way, such as setting environment
>>> variables, before executing the original test file.
>>> 
>>> It works well, and the DBIT has adopted the same approach.
>>> 
>>> I think the approach is good and more widely applicable than just the DBI.
>>> 
>>> I'd like to propose that we factor it out into a separate distribution.
>>> 
>>> Thoughts?
>> 
>> 
>> I had the same idea - even several separate distributions, but I rejected
>> myself there for following reasons:
>> 
>> 1) The variants of the driver result in an n choose k combination
>> 2) The driver attributes (eg. list of dbm backends, list of mldbm serializers)
>>   result in a cross product of those vectors.
>> 
>> When making separate distributions, 1st approach would be to create a really
>> generic combinatorics distribution. There are some, but all currently have
>> drawbacks. This distribution would likely develop an own life (combinatoric
>> problems tend to become complex and require lot of performance speed ups).
>> 
>> Based on that Math::Fast::Combi (*gg*) Test::MakeVariantTesTFiles would
>> require exactly to know what shall be done where.
> 
> I think it just requires a callback to prune combinations that aren't wanted.

That callback can be complicated and the decision can be made at several points:

* configuration definition
* test case
* datasource configurator (DSN::Provider, Test::Database)

The routing of those decision callbacks could be somewhat complicated but
with some effort reasonable.

>> I think - from the current point of view, this would result in an over
>> complex API. Maybe sourcing out the combinatoric algorithms would clean
>> up the code a bit and later when we know better we can additionally create
>> a distributions for computing variants (combinations, variations, permutations,
>> cross products, …) and based on that ?
> 
> I think a clean simple API could be designed. Keeping the test rewriting
> logic separate simplifies the rest of DBIT. It's a Separation Of Concerns.
> 
> Here's a rough suggestion for an API:
> 
>    write_variants(
>        # required:
>        test_files => [
>            't/foo.t', ...
>        ],
>        variants => {
>            wibble => { ... },
>            wobble => { ... },
>        },
>        # optional:
>        filter => sub {
>            my ($test_file, $variant_set) = @_;
>            delete $variant_set->{wibble} if ...;
>            return [ %$variant_set ]
>        }
>    );
> 
> The write_variants sub would call the filter callback for every combination
> of variants (in this example: wibble alone, wobble alone, wibble + wobble).

And for "none" ;)

> The filter can return zero or more array refs where each array contains
> the variants to be applied, in order. Returning nothing means don't
> create variants for that test. Returning multiple items makes it easy
> for the filter to generate extra permutations if appropriate for some tests.
> 
> What's missing?

Short answer: More levels and driver tunable settings. I give you a longer
example call (of the currently supported and implemented variants - beside
the filter callback - DBI::Test::Conf can populate):

    write_variants(
	# required:
	test_cases => {
	    foo => '@BEGIN@
do "t/foo.t"
@END@',
	    bar => '@BEGIN@
do "t/bar.t"
@END@',
	},
	variants => {
		     dbi => {
			      pureperl => {
					    name         => "DBI::PurePerl",
					    category     => "dbi",
					    cat_abbrev   => "z",
					    abbrev       => "p",
					    init_stub    => [ '$ENV{DBI_PUREPERL} = 2', ],
					    cleanup_stub => ['delete $ENV{DBI_PUREPERL};'],
					  },
			    },
		     dialect => {
				  ansi_sql => {
						name         => "ANSI SQL",
						category     => "ansi",
						cat_abbrev   => "l",
						abbrev       => "a",
						init_stub    => [ '...', ],
						cleanup_stub => ['...'],
					      },
				},
		     proxy => {
			 gofer => {
			     name       => "DBD::Gofer",
			     category   => "dbi",
			     cat_abbrev => "z",
			     abbrev     => "g",
			     init_stub =>
			       [ q{$ENV{DBI_AUTOPROXY} = 'dbi:Gofer:transport=null;policy=pedantic'}, ],
			     cleanup_stub => [q|delete $ENV{DBI_AUTOPROXY};|],
			 },
		     },
		     sql_engine => {
				     nano => {
					       name         => "DBI::SQL::Nano",
					       category     => "dbi",
					       cat_abbrev   => "z",
					       abbrev       => "n",
					       init_stub    => [ q{$ENV{DBI_SQL_NANO} = 1}, ],
					       cleanup_stub => [q|delete $ENV{DBI_SQL_NANO};|],
					     },
				   }
		    },
	drivers => {
		     dbm => {
			      category   => "driver",
			      cat_abbrev => "d",
			      abbrev     => "d",
			      driver     => "dbi:DBM:",
			      variants   => {
					    mldbm => {
						       f => { dbm_mldbm => 'FreezeThaw' },
						       d => { dbm_mldbm => 'Data::Dumper' },
						       s => { dbm_mldbm => 'Storable' },
						       # … JSON, YAML
						     },
					    type => {
						      s => { dbm_type => 'SDBM_File' },
						      g => { dbm_type => 'GDBM_File' },
						      d => { dbm_type => 'DB_File' },
						      b => {
							     dbm_type           => 'BerkeleyDB',
							     dbm_berkeley_flags => '...'
							   }
						    },
					  },
			      name => "DSN for DBM",
			    },
		     csv => {
			      category   => "driver",
			      cat_abbrev => "d",
			      abbrev     => "c",
			      driver     => "dbi:CSV:",
			      variants   => {
					    type => {
						      p => { csv_class => 'Text::CSV' },
						      x => { csv_class => 'Text::CSV_XS' },
						    },
					  },
			      name => "DSN for CSV",
			    },
		   },
	# optional:
	filter => sub {
	    my ( $test_file, $variant_set ) = @_;
	    delete $variant_set->{wibble} if ...;
	    return [%$variant_set];
	}
    );

Cheers
-- 
Jens Rehsack
pkgsrc, Perl5
[email protected]