Re: Quo Vadis JUnit

There are a lot of opinions, and great ideas for a new flexible testing framework going on here. I’d also like to give my five cents:

I was playing around with the new features of Java 8 and when it came to lambdas I started to think about how useful they are when it comes to testing. I always wanted to exercise how it would be to create a testing framework that is tested by itself. So after visiting the XP Days this year, I started to sit down and to play around with it.

When I started I made 3 clear assumptions:
1) I don’t want to write any kind of assertions, at least not those that come for free with Hamcrest.
2) I do want to use as much of pure Java as possible, especially as lambdas add a hole level of expressiveness
3) I wanted to place something like a global application context to configure the test system and allow plugins to register themselves for the test run


While number 1 was very easy to pick up, I was wondering how to express tests such that they read much nicer, but still by having pure Java code. I came up with the following rules:
a) Use Java classes to create test classes
b) Use inner Java classes annotated by @Context to create hierarchical contexts (like supported by the hierarchical context runner, see https://github.com/bechte/junit-hierarchicalcontextrunner/wiki)
c) Use Java methods annotated by @Test to represent tests
d) All types of Tests (groups, hierarchies, methods) implement a single interface, called Testable
e) The Testable has methods to get a name, a unique identifier and to run the test returning a TestResult object
f) Use a TestableFactory to create the Testable objects by handing in any kind of Object (a class, method, a name, etc.)
g) Allow plugins to register themselves with the TestableFactory to allow them to participate in the process of finding a Testable
h) This also allows plugins to load any kind of "type“ of tests, from any kind of location (could be a Java class, a text-file, or what have you…)
i) Use a global application context to load all the configurations which are provided by annotation on the test class and loaded by the framework
j) Instead of Rules, use Java8 lambdas to express these kind of concepts (see examples below)
k) Use default methods on interfaces to express global/shared concepts among classes / test cases


From these points, I would extract the following requirements:
- Test names: Within Java, they should conform to the Java Naming Conventions. Outside, it could be anything the according plugin parser can evaluate. :-)
- Runners: Runners should not be limited to one per class. Indeed, the TestableFactory would allow to have multiple runners for a test, such that you can have a class that runs its methods as tests and resolves additional tests from a test-file or so. Think of it as possibility to have your setup and teardown in Java methods while the tests cases are specified at another file.
- Reporting: Runners should notify Listeners during the test run and return a TestResult to the framework, which should gather all results and report them to Reports at the end of the run. These two concept allow mostly every kind of reporting that might be necessary, I think. We should get rid of these kind of Description thing and there should be no need to create a "tree structure“ of the tests and their children right before the start. I think the runners should return their results and it may be combined into a tree like structure in order to represent hierarchies of tests.
- Moreover, the TestResult class should be as abstract as possible. When talking about "Jump to Source“ we should think of it like an URL. The source of a Testable could be almost everything, like a class, a method, a txt-file or cucumber test, or what have you. Therefore, Jump to Source should point to a unique name, an unique identifier that must be resolved by the IDE, that wants to link back to the code / file / etc. Therefore, use the default naming conventions and schemas, which should reflect almost everything which is needed.
- Rules: There is nothing like a Rule needed. Global startup / initialization / teardown can be performed by the hook within the application context, once for a suite / test class / whatever. Rule for handling exceptions, run in parallel, run multiple times, or to test timing behavior can be easily represented by the lambdas mentioned in j). Here are two examples of how this might read (you can find it at https://github.com/bechte/JUT/tree/master/src/main/java/de/bechte/jut/matchers)

Example 1: Exception Handling
========================

@Test
public void someExceptionTest throws Exception {
  expectException(MyException.class).withMessage("Some error message!“).in(() -> {
    // here goes your test code without exception handling
    service.call("whatever“);
  });
}

Example 2: Run multiple times (parallel):
===============================

@Test
public void multipleRunTest throws Exception {
  runMultipleTimes(10).inParallel().withTimeout(1, TimeUnit.SECONDS).run(() -> {
    // here goes your long running test code
    service.call("whatever“);
  });
}

- Filtering and Sorting: Filtering and Sorting should also be configured during startup / bootstrapping the tests. The filters and sorters should be registered within the application context and should be used by the TestableFactory when searching / resolving the Testable objects. This is the perfect place to perform both, filtering & sorting.


I’ve put together all that I have in my little prototype and I really think it is worth to take a closer look at it: https://github.com/bechte/JUT. Please don’t take it the wrong way, this is just my ideas expressed by code, as I think it is much easier to understand some of the parts by looking at the code.

I hope some of the ideas make sense and will help to create a great new version for Testing in Java. :-)


Btw. I really think that Java 8 comes with so many new features and that the programming model has significally changed, such that it would be a great idea to "rethink“ the former ideas of what Unit Testing was all about and I wouldn’t be to sad if we start all over and do not support older versions with whatever will be the "new“ framework. The changes in the Java language really allow to break downward compatibility when creating something much better, I think.

All the best,

Stefan


[Non-text portions of this message have been removed]



------------------------------------

------------------------------------