A part of our team’s “definition of done” is having unit-tests in place and, unofficially, a minimum of 80% code coverage. Our Maven-based build process runs the tests and then creates code coverage reports in HTML format, that we can then consult in the documentation that Maven generates. Along with other reports, this helps us get a clear picture of where our code is in terms of stability and quality.

This is all great ; but when you’re in the middle of a task and you write unit-tests, it’s quite tedious to run the build and then open the HTML coverage report just to monitor your code coverage. I spend about 80% of my time for a task in Eclipse : opening and activating tasks with Mylyn, coding, writing unit-tests, running unit-tests and so on. This is why for me it made sense to seek out a way to monitor the code coverage in Eclipse.

After a bit of googling, I found a Java code coverage plug-in for Eclipse : EclEmma. As you can see from its name, it’s based on the EMMA Java code coverage tool. Here’s a list with its main features :

• a coverage mode in which applications launched or unit tests are instrumented and measured
• coverage overview : a coverage view containing a report on the source code coverage values at project-level, package-level and class-level
• source highlighting in the Java code editor using customizable colors
• customizable coverage counters
• multiple coverage sessions and session merging
• importing EMMA coverage data files
• exporting to EMMA coverage data, XML and HTML

The easiest way to install it is through the update site : http://update.eclemma.org. After the installation, you will notice a new launch mode appearing in the Eclipse toolbar, called Coverage, similar to the Run and Debug modes. This new mode allows you to run coverage reports on applications or unit-tests just like you would before run those applications or unit-tests.

In a project I’m currently working on, our server-side unit tests are written using TestNG. From Eclipse, I can run one or multiple TestNG units using the Eclipse TestNG plug-in, so I can easily verify that my code passes the unit tests. I have defined a launch configuration for each server-side project which runs all of the unit-tests. To check out the code coverage for those tests, all I have to do is create a coverage configuration and make sure that I select the source code folders to be instrumented.

I created a sample Java-Maven-TestNG project and added to it a simple class called ShoppingCartImpl along with a TestNG test class. Here’s how it looks :

Source code of a very basic shopping cart

As you can see, this is a very basic class. Now on to configuring the coverage settings ; this is a simple matter of clicking on the Coverage button in Eclipse’s toolbar and selecting the menu option Coverage Configurations…. This opens up the coverage configuration window, as seen below :

Creating a new EclEmma coverage configuration

All I have to do is select the TestNG test suite that I want to run and check the source folders that are relevant to code coverage. I’m using Maven and all my source code is in src/main/java so I only select that folder. I click Apply and then I can finally run the coverage report. I click on the Coverage button and theTestNG configuration is executed and the coverage report is available :

Code coverage report after a first run

In the bottom of the Eclipse window, you get a clear picture of the code coverage. As you can see, the are reports at project, package and class level, which also show up in the package explorer, in the left. To enable the decorators in the package explorer, go to the Eclipse menu and select Preferences -> General -> Appearance -> Label decorations, then make sure that the Java Code Coverage label decoration option is checked.

Another interesting feature of EclEmma is that after the code coverage instrumentation you can actually see the coverage in the source code. As you can see, each line relevant to the coverage report is marked with a color. Green is for 100% branch coverage, yellow is for some branch coverage and red for no coverage at all. The shopping cart has quite a low code coverage so I did shape it up. After a bit of fiddling with the code, I get a 100% code coverage and a very nice report :

The ShoppingCart class with 100% code coverage

When you write developer tests, you start with just a few and as your code base gets larger and larger, so does the number of tests. The build starts to take more and more time and soon you avoid running the build as often as possible. If you get to this, then you need to categorize your tests.

How do I do it ?

First, stop calling every developer test a unit test.

Second, categorizing your formerly-known-as-unit-tests tests means you separate them into different layers based on external dependencies, complexity, time of execution. There are 3 layers of developer tests : unit tests, component tests, system tests.

Categories

Unit tests

Unit tests are run against objects without any external dependencies. Any external dependency is mocked and the unit test concentrates only on the object you wrote. This is why unit tests are usually simpler to write and take less time to execute.

Component tests

Component tests validate multiple objects with external dependencies and their interaction. This means that you deal with databases, file systems, HTTP connections and so on, and with the interaction between your objects (the component) and the external systems. Component tests tend to be more complex than unit tests, they take more time to write and more time to execute.

System tests

When you want to test your application from one end to the other, you write system tests. They verify your application as if an user would use the application, that is they mimic one.

I have my categories, now what ?

Now it’s time to update your build process.

Update the build process

You have 3 categories of developer tests : unit test, component tests and system tests. Because unit tests run fast, you can choose to run them during each build and during each CI build. In contrast, since component tests and system tests take much more time to execute, you can choose to skip them during a regular CI build and run them less often.

Setting up a staged build

In his articles about Continuous Integration, Martin Fowler suggests to keep the build fast by setting up a staged build.
The basic idea is that by setting up a staged build (a 2 phases build) you can achieve a compromise between execution time and testing thoroughness. The developers need a feedback on their commit, so the testing process needs to be as thorough as possible, but the feedback must be delivered as quickly as possible because you don’t want to keep the developers waiting while the build is running.

To accomplish this, you can set up two builds : a commit build – the build that everyone executes before committing code – and a secondary build – a full build.

The commit build

The commit build has to be run by everyone before committing code. During this build, only unit tests are executed, so the build doesn’t take a lot of time ; on the other hand, this also means that the application is not tested at a higher level.
This build is also run by the CI system after each commit.

The secondary build

The secondary build is a full build, it runs all of the tests – unit tests, component tests, system tests. This means it takes a lot longer to complete, but on the other hand it gives you a complete feedback on your application.

The secondary build is run by the CI system. Depending on the time it takes to complete the secondary build, you can have the CI system to run it after each successful commit build or at regular time intervals.

Categorizing tests in Java

I use TestNG to write developer tests for my Java code. The reason for this is that it has some features that I haven’t found elsewhere yet, so I stick to it. One of these features is, coincidentally, the ability to define and use test groups. At its core, it’s all about being able to define groups of tests and to run one or more groups of tests during a testing session or during a build.

The approach I took on a project was to write the tests using TestNG and to divide them into three TestNG test groups, corresponding to the test categories I described earlier. First, I declared some constants for the groups :

public static final String GROUP_UNIT_TEST = "unit-test";
public static final String GROUP_COMPONENT = "component";
public static final String GROUP_SYSTEM = "system";

Then whenever I added a test method, I would assign it to a test group. For instance, unit tests :

@Test(groups = { GROUP_UNIT_TEST })
public void testAdaptNullList() {
...
}
@Test(groups = { GROUP_UNIT_TEST })
public void testAdaptList() {
...
}

Component tests would look very similar :

@Test(groups = { GROUP_COMPONENT })
public void testSqlConnection() {
...
}
@Test(groups = { GROUP_COMPONENT }, dependsOnMethods = { "testSqlConnection" })
public void testGetTargetData() {
...
}

In the code block above, you can notice another nice feature of TestNG : dependent methods. In the test above, the test method testSqlConnection is run first. If it fails, the dependent method testGetTargetData is skipped.
With TestNG’s dependent methods mechanism, you can have some methods from your tests depend on other methods to make sure a certain number of test methods have completed and succeeded before running more test methods. This feature makes sure you don’t waste time running certain tests which will fail for sure if other tests already failed.

And now for some system tests :

@Test(groups = { GROUP_SYSTEM })
public void testGetFilters() {
...
}
 
@Test(groups = { GROUP_SYSTEM }, dependsOnMethods = { "testGetFilters" })
public void testGetFiltersPeriods() {
...
}
 
@Test(groups = { GROUP_SYSTEM }, dependsOnMethods = { "testGetFilters" })
public void testGetFiltersCallCenter() {
...
}

Once the tests are in place, I can configure the builds accordingly. I can configure the commit build, based on Maven, to run only unit tests :

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
      <groups>unit-test</groups>
    </configuration>
  </plugin>

The secondary build, on the other hand, can run all the tests :

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
      <groups>unit-test,component,system</groups>
    </configuration>
  </plugin>

Easy, right ? TestNG is integrated nicely with Maven using the Maven Surefire Plugin.
TestNG is also integrated with Eclipse so that you can run tests or groups of tests from Eclipse while you’re coding. If you’re an Ant guy, check out the TestNG Ant task. And another thing if you’re an Ant guy : switch to Maven.