Categorizing tests in Java

When you write developer tests, you start with just a few and as your code base gets larger and larger, so does the number of tests. The build starts to take more and more time and soon you avoid running the build as often as possible. If you get to this, then you need to categorize your tests.

How do I do it ?

First, stop calling every developer test a unit test.

Second, categorizing your formerly-known-as-unit-tests tests means you separate them into different layers based on external dependencies, complexity, time of execution. There are 3 layers of developer tests : unit tests, component tests, system tests.

Categories

Unit tests

Unit tests are run against objects without any external dependencies. Any external dependency is mocked and the unit test concentrates only on the object you wrote. This is why unit tests are usually simpler to write and take less time to execute.

Component tests

Component tests validate multiple objects with external dependencies and their interaction. This means that you deal with databases, file systems, HTTP connections and so on, and with the interaction between your objects (the component) and the external systems. Component tests tend to be more complex than unit tests, they take more time to write and more time to execute.

System tests

When you want to test your application from one end to the other, you write system tests. They verify your application as if an user would use the application, that is they mimic one.

I have my categories, now what ?

Now it’s time to update your build process.

Update the build process

You have 3 categories of developer tests : unit test, component tests and system tests. Because unit tests run fast, you can choose to run them during each build and during each CI build. In contrast, since component tests and system tests take much more time to execute, you can choose to skip them during a regular CI build and run them less often.

Setting up a staged build

In his articles about Continuous Integration, Martin Fowler suggests to keep the build fast by setting up a staged build.
The basic idea is that by setting up a staged build (a 2 phases build) you can achieve a compromise between execution time and testing thoroughness. The developers need a feedback on their commit, so the testing process needs to be as thorough as possible, but the feedback must be delivered as quickly as possible because you don’t want to keep the developers waiting while the build is running.

To accomplish this, you can set up two builds : a commit build – the build that everyone executes before committing code – and a secondary build – a full build.

The commit build

The commit build has to be run by everyone before committing code. During this build, only unit tests are executed, so the build doesn’t take a lot of time ; on the other hand, this also means that the application is not tested at a higher level.
This build is also run by the CI system after each commit.

The secondary build

The secondary build is a full build, it runs all of the tests – unit tests, component tests, system tests. This means it takes a lot longer to complete, but on the other hand it gives you a complete feedback on your application.

The secondary build is run by the CI system. Depending on the time it takes to complete the secondary build, you can have the CI system to run it after each successful commit build or at regular time intervals.

Categorizing tests in Java

I use TestNG to write developer tests for my Java code. The reason for this is that it has some features that I haven’t found elsewhere yet, so I stick to it. One of these features is, coincidentally, the ability to define and use test groups. At its core, it’s all about being able to define groups of tests and to run one or more groups of tests during a testing session or during a build.

The approach I took on a project was to write the tests using TestNG and to divide them into three TestNG test groups, corresponding to the test categories I described earlier. First, I declared some constants for the groups :

public static final String GROUP_UNIT_TEST = "unit-test";
public static final String GROUP_COMPONENT = "component";
public static final String GROUP_SYSTEM = "system";

Then whenever I added a test method, I would assign it to a test group. For instance, unit tests :

@Test(groups = { GROUP_UNIT_TEST })
public void testAdaptNullList() {
...
}
@Test(groups = { GROUP_UNIT_TEST })
public void testAdaptList() {
...
}

Component tests would look very similar :

@Test(groups = { GROUP_COMPONENT })
public void testSqlConnection() {
...
}
@Test(groups = { GROUP_COMPONENT }, dependsOnMethods = { "testSqlConnection" })
public void testGetTargetData() {
...
}

In the code block above, you can notice another nice feature of TestNG : dependent methods. In the test above, the test method testSqlConnection is run first. If it fails, the dependent method testGetTargetData is skipped.
With TestNG’s dependent methods mechanism, you can have some methods from your tests depend on other methods to make sure a certain number of test methods have completed and succeeded before running more test methods. This feature makes sure you don’t waste time running certain tests which will fail for sure if other tests already failed.

And now for some system tests :

@Test(groups = { GROUP_SYSTEM })
public void testGetFilters() {
...
}
 
@Test(groups = { GROUP_SYSTEM }, dependsOnMethods = { "testGetFilters" })
public void testGetFiltersPeriods() {
...
}
 
@Test(groups = { GROUP_SYSTEM }, dependsOnMethods = { "testGetFilters" })
public void testGetFiltersCallCenter() {
...
}

Once the tests are in place, I can configure the builds accordingly. I can configure the commit build, based on Maven, to run only unit tests :

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
      <groups>unit-test</groups>
    </configuration>
  </plugin>

The secondary build, on the other hand, can run all the tests :

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
      <groups>unit-test,component,system</groups>
    </configuration>
  </plugin>

Easy, right ? TestNG is integrated nicely with Maven using the Maven Surefire Plugin.
TestNG is also integrated with Eclipse so that you can run tests or groups of tests from Eclipse while you’re coding. If you’re an Ant guy, check out the TestNG Ant task. And another thing if you’re an Ant guy : switch to Maven.