Re: preforking prove

Hi Jonathan,

I have just one question. You wrote that you're using Test::Class and "many of the tests start by loading a bunch of the same modules". That confuses me as Test::Class was originally designed to speed up test suites and did so by loading everything *once* in the same process.

Are you using a separate .t script per test class? That would cause the reloading. Otherwise, a single .t script loading all of your test classes (Test::Class::Load helps) should help you load all classes at once. Assuming they have a sane setup and all call their own test control methods (startup/setup/teardown/shutdown), you could drop something like this in your base class:

    # see also http://www.slideshare.net/Ovid/a-whirlwind-tour-of-testclass
    sub setup : Tests(setup) {
        my $test = shift;
        $test->reset_singletons;
    }

Yes, it's a hack to work around the fact that you have a bunch of singletons with mutable state, but it seems like this would be much easier (though perhaps less fun :) than your preforking solution.
 
Cheers,
Ovid
--
Twitter - http://twitter.com/OvidPerl/
Buy my book - http://bit.ly/beginning_perl
Buy my other book - http://www.oreilly.com/catalog/perlhks/
Live and work overseas - http://www.overseas-exile.com/


>________________________________
> From: Jonathan Swartz <[email protected]>
>To: perl-qa <[email protected]> 
>Sent: Tuesday, 6 November 2012, 2:03
>Subject: preforking prove
> 
>We have a large slow test suite at work (Test::Class, 225 classes, about 45 minutes run time). Many of the tests start by loading a bunch of the same modules. Obviously we could speed things up if we could share that loading cost.
>
>I'm aware of Test::Aggregate and Test::Aggregate::Nested, but a number of our tests run into the caveats (e.g. they change singletons -- yes, this is not ideal, but also not easily changeable).
>
>I thought of an alternative to Test::Aggregate called "preforking prove", or pfprove, which does the following:
>
>* Accepts the same arguments as prove
>* Preloads the module(s) in -M
>* Launches a Starman server in the background
>* For each test file, makes a request to the Starman server. The Starman child runs the  test and exits.
>
>The idea is that you preload all the common stuff in the Starman parent, so that forking each child and running each test is fast (at least on Linux).
>
>Potential advantages over Test::Aggregate:
>* runs one test per process, so avoids some caveats
>* keeps the TAP in traditional form (one TAP stream per file)
>* works well with parallel prove
>
>Potential disadvantages:
>* lots of extra complexity (requires Starman or similar, need to make sure it shuts down, need to handle errors, etc.)
>
>Curious what you think. Is there something like this out there already? Potential problems?
>
>If I do this I might call it Test::Aggregate::Preforking, just to keep it in the same category.
>
>Thanks!
>Jon
>
>
>
>