Re: [PHP-DEV] Concurrency support for run-tests

When building PFTT, the PHP Full Test Tool, one goal was maximizing speed
so that we would be more likely to frequently Fully test PHP (Full-coverage
of many scenarios), and to do that I built it to run tests concurrently...

Concurrent/parallel execution can cause breaks that otherwise work with
normal serial execution, so definitely don't enable concurrency in
run-tests by default.

run-tests.php's two main benefits are that it supports running the huge
library of 15k+pecl PHPTs (compatiblity) and its a lightweight tool so that
anyone who just built PHP can run `make test` to test their build (whereas
PFTT Is a heavyweight tool).

In making/merging major changes (like concurrency) the most important thing
is to not break that compatibility and dependability:

Your patch will fail without the /proc filesystem(mac osx?). On Windows,
you can check the %NUMBER_OF_PROCESSORS% environment variable. For unices
that don't have /proc there may not be a single simple way to check.


For better performance, PFTT runs a pool of 3 to 4 processes per CPU, when
the processes finish quickly, it runs fewer and when they slow down (due to
pending IO ops), it runs more processes.  See:
http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/CLIScenario.java;h=30c81aabcb56698eebe4884d915298c92332b50d;hb=refs/heads/master#l61
and for other SAPI scenarios like Apache it runs 2 or 3 processes, see:
http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/ProductionWebServerScenario.java;h=d509bddda2b3756f05321ddc273600aeb38372be;hb=refs/heads/master


For more speed, rather than grouping all, fe open_basedir tests into a
single `Concurrency Group` its better to have them all operate in separate
temporary directories and database tables (then you could run 32 or 64
openbase dir tests at once).

The ext/standard/tests/file and ext/mysql* PHPTs take the longest to run
and mostly do IO operations so they benefit the most from maximizing
concurrency.

For databases, PFTT will create a new database (and for phpunit tests
populate the database) for each test runner thread, See:
http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/MySQLScenario.java;h=f768f58c4494cb2705a571ece22d533bfdc15192;hb=refs/heads/master#l164




On Wed, Sep 10, 2014 at 3:36 AM, Michael Wallner <[email protected]> wrote:

> On 09/07/14 22:00, Stas Malyshev wrote:
> > Hi!
> >
> >> Having thought a bit about the subject, the heuristic I came up with
> was to
> >> 1) do not run tests from the same extension in parallel, and 2) do not
> run
> >> tests from certain extensions in parallel (e.g., mysql and pdo_mysql).
> >
> > How about making it a whitelist? For tests/extensions we know it's easy
> > to paralellize (like ones not having external dependencies/servers/etc)
> > simple script could whitelist a lot of them. For those more tricky, we
> > could whitelist them gradually.
> >
>
> So how about adding the concurrency switch, but leaving it disabled by
> default?
>
> --
> Regards,
> Mike
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>