Re: Test::Database and Test::DSN

On Fri, Apr 04, 2014 at 12:32:10PM +0200, Philippe Bruhat (BooK) wrote:
> On Thu, Apr 03, 2014 at 03:22:56PM +0100, Tim Bunce wrote:
> 
> > Some logic to assist with table naming and cleanup etc will be
> > developed as part of DBIT and could be extracted. If so, I think it
> > belongs in a separate distro. (Separation of Concerns etc.)
> 
> And if Test::Database only deals with giving out Test::DSN objects, then
> it should not need to care about naming issues (except for file-based
> DSN, but that's a simpler problem).
> 
> Does the logic already exist somewhere as code or module?

> > > Another option is that Test::Database could provide a function that
> > > returns a unique prefix for a given script, so that the test script
> > > can use that when creating the tables. That would provide isolated
> > > "namespaces", and then we don't have to deal with locking directly.
> > > With that namespace, we can even let Test::Database do things like
> > > "drop all tables in that namespace".
> > 
> > That's the approach DBIT is taking. There's a basic start in here:
> > https://github.com/perl5-dbi/DBI-Test/blob/master/sandbox/tim/lib/DBI/Test/FixtureProvider/GenericBase.pm#L49
> > I'd be very happy to see that factored out into a separate distro.
> > I'd rather it wasn't shipped within Test-Database though as I think it
> > has wider uses.
> 
> OK. Do you have a name in mind already? The namespace:: top-level seems
> to be mostly about Perl namespaces.
> 
> It seems to me that it's something that should have various "strategies"
> to generate a "namespace". Like, which character set is available,
> are there any length limits, that kind of thing. The constraints are
> different for table names, database names, etc.
> 
> And I'll expand by just copying and pasting the comments from
> DBI::Test::FixtureProvider::GenericBase:
> 
>     # could also provide methods to:
>     # - supply a regex that matches the get_dbit_temp_name name
>     # - parse a get_dbit_temp_name to extract especially the yymmddhh
>     # - drop all get_dbit_temp_name tables older than N hours (0=all)

I think all that's needed, at least for now, is something like:

    $np = NameProvider->new(style => 'basic', prefix => 'dbit');
    $name = $np->generate_new_name;
    my @names = grep { $np->is_generated_name($_) } @names;
    $regex = $np->get_generated_name_regex;
    $hash = $np->parse_generated_name($name); # eg to get date etc

I expect the format that DBIT is planning to use (see link above) would
suffice for all the databases we're currently targeting.
(Test::Database could support alternative providers via a config option
to set the name of the style, and/or the name of the provider class.)

I'll do some prototype work and start a new thread with a more concrete
proposal, including a class name :)


> >     dbi:DBM:f_dir={CREATE_DATABASE}
> >     dbi:Oracle:sid={CREATE_DATABASE}
> 
> I really like this. The notion of "private database" is both very explicit
> and nicely generic.
> 
> So after having read the config files, Test::Database has:
> - a list of plain DSN
> - a list of DSN that can be used by the private database creation service
>   to produce a DSN on demand (similar to the current driver_dsn)
> 
> I think that the warning you gave above ("the database you get connected
> to may be in active use by other applications, and/or concurrent instances
> of your application, so you need to take appropriate care.") only applies
> to the first kind of DSN.

Yeap. Though distros using Test::Database won't know what kinds of DSN
they'll get so they should assume the worst and be defensive.

> So, for DBD::CSV, a dsn for the private database creation service would
> look like this:
> 
>    dbi:CSV:f_dir={CREATE_DATABASE}
> 
> And following Tux's preferences, it could also be:
> 
>    dbi:CSV:f_dir={CREATE_DATABASE};f_ext=.csv/r
> 
> Now there are plenty of combinations of parameters that make up valid but
> slightly different DSN. I think that Test::Database should only provide
> one of those by default (to avoid combinatorial explosion), to be dropped
> if the configuration happens to provide one of more DSN for that DBD.

I don't follow you here BooK.

> So, for all supported DBD, Test::Database would provide a private
> database factory in the form of a DSN with the {CREATE_DATABASE}
> placeholder. And actually, that's really something that each
> Test::Database::Driver should provide.

Yes, assuming I'm understanding you correctly.


> For the "private databases", we can even provide various levels of privacy
> (as offered by the namespace-providing module), like DSN that are private
> to the project (the namespace is generated using the project top-level
> directory), or to a given test script (the namespace is generated
> using the script full path). This is something that should be part of
> the request done by the test script:
> 
>     # this DSN is expected to not be shared with anything else
>     @handles = Test::Database->handles( privacy_level => 'script' );
> 
>     # this DSN will be shared across test scripts in the same project,
>     # enabling some 000-startup.t to setup some data to be used by later
>     # scripts, for example
>     @handles = Test::Database->handles( privacy_level => 'project' );
> 
> I'm not sure which privacy level is a good default. Probably script-level.

Umm. That might be over-engineering at this stage. 'project' level could
be added later if there's a clear need.


> The best location for private file-based DSN is probably cwd itself,
> under some hidden directory.

Why hidden?

> So a .test-database/ directory at the project level would be the home
> for the project local configuration file, and the various file-based
> private databases created for the project.

Just "test-database" seems fine to me.


It's great to see this moving forwards. Thanks BooK.

Tim.