| Newsgroups |
gmane.comp.java.junit.user |
| Message-ID |
<[email protected]> |
I'll go over the topics on by one, adding some more related topics. For
some of them I've implemented a solution in the Jumi test runner
(http://jumi.fi/) in which case I will point out how I did it there.
## Test names ##
I think test names should allow any characters, including space and
punctuation, so that they could be written as sentences describing the
desired features. At most newline may be disallowed.
## Test identification ##
A concern related to test names is test identification. In Jumi test
names do not have to be unique. For test identification it uses
hierarchial numeric IDs
(https://github.com/orfjackal/jumi/blob/master/jumi-api/src/main/java/fi/jumi/api/drivers/TestId.java)
which also signify the order in which the tests should be shown in test
results as a tree. Those IDs are not shown to the user and since the IDs
may change between test runs (e.g. if reflection returns test methods in
different order), using them for test filtering might not work, but
filtering would work better based on test names.
## Multiple execution/cascading names ##
In Jumi tests may be executed multiple times and a test's execution may
contain the execution of other tests as sub-steps, in an arbitrarily
nested structure. This supports all the "BDD frameworks" which often use
nesting for organizing tests. See
https://github.com/orfjackal/jumi/wiki/Execution-Model-and-Terminology
## Global Rules ##
I don't know how this could be fully solved without tying it to the
details of a particular testing framework. Even with JUnit 4 each
org.junit.runner.Runner implementation is responsible for handling the
rules itself. It may be hard/impossible to create global rules that wrap
the individual test methods of any testing frameworks.
As a limited solution, rules at the level of the whole suite or
individual test classes should be possible. For example a rule which
starts up the application/databases once before all tests and shuts it
down after all the tests are finished. Or a rule that wraps around the
execution of a test class, for example to set a timeout.
I wonder how it should be declared that which global rules to use.
Probably explicitly listing them in a configuration would be best.
Classpath scanning would make it too unpredictable and it would delay
the first test's execution until after the scanning is finished.
## Finding Tests ##
Agreed, there should be a common way for doing this to avoid minor
differences between tools. That's why also in Jumi I've implemented test
discovery as a built-in feature. It crawls though the compile output
directories and looks for files which match a regex or glob pattern.
Test execution happens in parallel with test discovery, starting
immediately when the first test class is found.
Using naming patterns has the added benefit that it's not tied to .class
files, but may be used by frameworks which store tests in text files
(e.g. Cucumber and other BDD frameworks) or in other programming
languages (e.g. Clojure).
## Finding Tests Incrementally ##
In some testing frameworks where test are declared as method calls (by
passing in the test name and a closure), especially with nested tests,
it's not possible to find out a list of all tests before executing any
tests. This is one of the reasons why I created the Jumi test runner,
because in the Specsy testing framework (http://specsy.org/) I had to
use hacks to work around the JUnit 4 test runner's assumption of knowing
what tests exists before executing them (otherwise some IDEs behave
weirdly).
## Jump to source code ##
Since there are lots of different ways of declaring tests, currently
IDEs need to have custom support for each testing framework. I've been
thinking that it might be possible for a testing framework to report to
the test runner the location of each test (i.e. file and line number),
so that IDEs would not need to figure it out themselves.
For JUnit 4 style testing frameworks, this could be done by reading line
numbers from the bytecode of the test methods. For testing frameworks
where tests are declared by passing the test name and a closure as
arguments to a method, this could be done by creating an Exception and
reading the line number of the calling method from the stack trace elements.
## Filtering ##
Related to the above discussion about test identification, I think using
test names is the most reliable method of filtering tests, with the
(minor) caveat that running a single method may not be possible if two
tests have the same name in the same class.
Related to the above discussion about cascading and nested tests, I
think that test filtering will need to take into account the test
hierarchy. So instead of just a single filtering pattern (regex?), there
would need to be a list of them, one for each depth level of tests.
I have some doubts whether it will be possible to create a filtering
mechanism that fits every testing framework. Some frameworks may be able
to support the filtering mechanism only partly. Also IDE support may
have to vary between testing frameworks - in the general case the IDE
will know what tests exists only after the tests have been run once,
i.e. it would not be possible to select inside the editor "run only this
test" unless the IDE has custom support for the testing framework in
question.
## Test Groups ##
Related to filtering and finding tests is test groups/categories. For
example we want to make it easy to run just unit tests without
integration tests. I think a good solution would be the ability to give
textual tags for test classes, and then the ability to specify
expressions such as "!db & !jms" to include/exclude tests with a
particular tag.
## Order of Tests ##
I think this is a very testing framework specific aspect - how to
declare the execution order or dependencies between tests. When
everything is single-threaded, then the testing framework should already
be in full control of the order of executing things. But when parallel
test execution is added to the mix, some support from the test runner
might be needed, for example to say that some test classes should not be
run in parallel (because of e.g. a shared database).
For Jumi I'm planning on implementing support for restricting the
parallelism by annotating test classes (currently everything is in
parallel). I'm thinking about having three options: (1) run each test
method in parallel, (2) run tests methods in this class sequentially but
allow other test classes to be run in parallel with it, (3) run this
test class sequentially and don't run anything else in parallel with it.
I think #1 and #3 will at least be needed, but for #2 I'm wondering
whether it should be made more generic by using test groups, for example
"don't run tests tagged with 'db' in parallel with any other tests
tagged with 'db'".
And as to what comes to execution order within one test class, for
example TestNG's dependent methods
(http://testng.org/doc/documentation-main.html#dependent-methods), I
think Jumi is already generic enough: There the testing framework starts
test method executions by submitting them to an
java.util.concurrent.Executor and it is possible for the testing
framework to delay submitting some test to the Executor until all the
tests it depends on have finished executing.
--
Esko Luontola
www.orfjackal.net
------------------------------------
Posted by: Esko Luontola <[email protected]>
------------------------------------