Skip to content

Experimenting with JUnit Categories

by on August 30, 2012

Update 2013-03-20: JUnit moved from Kent Becks GitHub account to that of junit-team. Links updated in the post.

JUnit Categories are an experimental feature to group tests despite from their name or package. You can even configure the Maven Surefire and Failsafe plugin to select tests by those categories using the groups and excludedGroups configuration. Today I tried to build some sophisticated categories to categorize our tests. Categories I created for example (I skipped the typical “Test” appendix as this seemed to verbose to me:

  • Firefox – for tests which only work on Firefox
  • InternetExplorer – for tests which only work on Internet Explorer
  • AnyBrowser – subclassing Firefox and InternetExplorer
  • WithinMilliseconds – for very fast tests
  • WithinSeconds – for fast tests (also extends WithinMilliseconds)
  • WithinMinutes – for slower tests (also extends WithinSeconds)
  • Stable – for tests which never fail
  • Unstable – for tests which are unstable, thus fail from time to time

Excursus: I favored WithinSeconds and WithinMinutes over typical categories Fast and Slow because typically everyone will understand something different for what a fast and what a slow test is. And even for a type of test this might vary. A unit test which takes seconds is considered to be slow. While an integration test taking seconds might be perceived as being fast. Thus using a time unit here seems to be much more convenient.

And back again… Having those categories I wanted to create a suite which runs all fast and stable tests which run on any browser. Those tests should be selected for an initial test of a Web UI before running longer running tests or those tests which are only working on specific browsers.

So I started to write the test suite using JUnit 4.10:

@RunWith(Categories.class)
@Categories.IncludeCategory(AnyBrowser.class)
@Categories.IncludeCategory(WithinSeconds.class)
@Categories.IncludeCategory(Stable.class)
@Suite.SuiteClasses(CategorizedTest.class)
public class FastWebUISuite { }

Actually you might have observed the bug: You cannot use the IncludeCategory annotation twice. But JUnit’s API does not provide an array of values. And even if this would be provided… what is the boolean operator? OR or AND? For Maven Surefire/Failsafe Plugin you can specify more groups as comma separated list. They are ORed.

To run all tests as JUnit Suite you have to create an extra category:

public interface StableFastAnyBrowser extends Stable, AnyBrowser, WithinSeconds {}

But you are required now to update any test code to also have the category StableFastAnyBrowser. A test having the categories Stable, AnyBrowser and WithinSeconds won’t be selected.

Now let’s try a naïve approach using Failsafe configuration:

<groups>AnyBrowser,Stable,WithinSeconds</groups>

Of course this will execute too many tests as the groups are ORed. You can also start with exclusions and might come closer to the result. But all in all you will experience that for a fine grained category tree the expressions to select those tests are too weak. What you would need, so my assumption, is some kind of query language to select those tests.

I don’t know TestNG too well. It also offers a concept of groups. It is at least by far better documented than the JUnit categories. And it has a little richer selection logic as you might use regular expressions to match group names.

Conclusion

At least for a fine grained selection of tests through a bunch of categories the existing solutions (JUnit as well as TestNG) are too weak. But if you stick to a simple Category scheme like for example one Category for all tests you would like to run prior to all other tests then you might be satisfied with the existing possibilities of categories.

Links

From → Testing

2 Comments
  1. John permalink

    This is cool. The test profiles feature is what makes the Categories concept much better. Since there are very few such articles on categories feature of Junit, I am quoting another one I stumbled upon: test case categories

Trackbacks & Pingbacks

  1. Design of Test Categories « Minds

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s