Overview

BugUnit extends JUnit with the concept of a 'bug' and provides various useful utilities when dealing with a large number of long running tests. Simply add the notion of a 'bug id' to your JUnit test code:
  BugAssert.assertEqualsBug("TST-6745","name incorrect","pete",name);
and an HTML bug report will be generated that cross-references bugs with entries in your bug tracker:

With a JIRA bug tracker, automatic state synchronization is supported. If all tests for a bug succeed, the bug is set to 'resolved'. If at least one test fails, the bug is set to 'open'. BugUnit facilitates continuous integration as it prepares Junit tests to integrate better with continuous build systems such as Maven, Cruise Control and your bug tracker.

The generated Bug Report shows for every bug:

In addition to this bug-specific information, the report also includes a set of deadlocking tests methods and a summary of open and closed bugs. Based on a configuration option, the report can also track standard JUnit failures and include them in this bug report.

Besides this 'bug-awareness', BugUnit includes the following optional utilities to facilitate functional testing:

Getting Started

After unpacking bugunit-1.0.tar.gz into a directory of your choice, you will find a default bugunit.properties file and the following directories:

To run the examples, follow these steps:

  1. create a project in the base directory
  2. add BugUnit jars to your CLASSPATH:
      if your are planning to use automatic JIRA updates: add junit-3.8.1.jar and all libraries in the <BugUnit>/lib directory
      otherwise: only junit-3.8.1.jar and <BugUnit>/lib/bugunit-1.0.jar are needed
  3. run any of the examples as JUnit Test.
The examples will then use the configuration in bugunit.properties and generate a bug report in: samples/bugreport.html

BugUnit Configuration

All BugUnit parameters are configured through a bugunit.properties configuration file. To debug the operation of BugUnit, enable debugging roles through the bugunit.debug parameter. To enable all roles, set it to: "BugUnit,Monitor,Tracker,File,Process,Runner,Jira".

Writing a bug test with JUnit

One problem with standard JUnit testing is its unawareness of bugs. To highlight this issue, let's take the common scenario where someone finds a new bug and enters it into your bug tracker, giving it a new id, such as "TST-6745".

A good practice (using a test-first approach) is to write a new JUnit test exposing this bug:

public class ExampleJUnitTest extends TestCase
{
  public void testValueNotRight()
  {
    ... get the value through the method to test ...
    assertEquals("testing TST-6745, value incorrect.","pete",value);
  }
}
In an ideal world, each bug is fixed before the next release of the software.

In reality, however, some bugs are of  lower priority and don't need to be fixed immediately. Especially, with a continuous build process, the next release could only be days away. For some bugs you could have made the decision that a fix to these bugs will only come in some future release. On the other hand, continuous build system (such as Cruise Control) or your build manager (such as Maven) will require that all tests succeed before even continuing with additional packaging steps. With standard JUnit, a system is considered 'broken' as soon as one test fails.

What to do ?

One alternative would be to comment out thee bug-assertions or changing assertEquals() to assertNotEquals(). But will you remember to change it back, once the bug is fixed ? This is good test code. Why artificially change it ?
Or you could simply get used to the fact that some of your hundreds or thousands of tests fail. But then, will you recognize that you just discovered a new bug when your number of JUnit failures climbed from 47 to 48 ?

With JUnit as used above, there is also no central place to check which bugs are currently 'open' and which are 'closed'. Since bugs are more coarse grained than JUnit failures (a single bug in the system can cause many JUnit assertions to fail), JUnit alone could never solve this problem.

Writing a bug test with BugUnit

BugUnit adds 'bug-awareness' to JUnit. When writing a test to expose a bug, you should write a 'bug test' as shown below. All JUnit assert methods (and some more) are available by inheriting your test case from BugTestCase instead of Junit's TestCase. BugUnit's assert statements add a new argument to the assertions. This first argument is of type String ("TST-6745" for example) and denotes the bug id that this assertion is testing. If this assertion fails, the bug is open. If all assertions for this bug id succeed, the bug is considered fixed. Multiple assertions can test the same bug.
public class ExampleTest extends BugTestCase
{
  public void testNameNotRight()
  {
    ...
    assertEqualsBug("TST-6745","name incorrect","pete",name);
  }
}
Instead of inheriting your test case from BugTestCase, you could also use the static methods of BugAssert as shown below. This approach is less intrusive as it does not require to change your existing TestCase type hierarchy.
public class ExampleStaticTest extends TestCase
{
  public void testNameNotRight()
  {
    ...
    BugAssert.assertEqualsBug("TST-6745","name incorrect","pete",name);
  }
}
Running such a BugUnit-enabled test will generate an issues.xml file that contains all stack-traces of failing and succeeding bug tests. At the end of a test run, the data model of issues.xml is passed to a set of pluggable Reporters. These reporters generate a Text or HTML  Bug Report or can automatically update your bugtracking system (e.g. JIRA).

When running a large number of tests automatically, it can further be beneficial to ignore open bugs. To let JUnit ignore all open bugs, set the bugunit.openbugs.throw parameter to false. In this case, open bugs will not throw a JUnit AssertionError and Cruise Control or Maven will continue assuming a 'sane' system. The JUnit report will then only show newly discovered problems. Known bugs are shown in BugUnit's  Bug Report.

Once a bug is fixed, you might want to be notified, also in JUnit, if it breaks again. To achieve this, you can use the assert..BugFix() instead of the assert..Bug() methods. These BugFix assertions will throw JUnit errors even if bugunit.openbugs.throw was set to false.

public class ExampleBugUnitTest extends BugTestCase
{
  public void testNameNotRight()
  {
    ...
    BugAssert.assertEqualsBugFix("TST-6745","name incorrect","pete",name);
  }
}

Tracking of bugs can be accumulative or not. Accumulative bug tracking (bugunit.tracker.accumulate=true) is useful when you iteratively explore your test cases. Initially you should remove issues.xml and then run a group of test cases, for example in your IDE. Then run another group of tests. With accumulative testing, BugUnit will add new bugs to the previously found ones in issues.xml. The current bug report will contain all bugs since issues.xml was last deleted. This setting is useful for manual and iterative exploration of tests.
With bugunit.tracker.accumulate=false, BugUnit will overwrite issues.xml with every test run. This setting is useful when you are running all tests in one go, for example through Maven, and want to start with a fresh issues.xml file.

To disable the tracking of bug tests, set bugunit.tracker.trackbugs to false. Issues are no longer stored to issues.xml and your test software will behave as if you were using standard Junit assertions. The bug ids in the bug assertions are ignored completely. This setting is useful to temporarily disable BugUnit's 'bug-awareness'.

The Bug Report

The default reporter implementation is a HtmlReporter which generates a bug report such as this.

To write a new reporter, extend the Reporter interface and add its classname to the bugunit.reporter.classes parameter. bugunit.reporter.classes contains a comma-separated list of reporter class which are invoked in the specified order.

The HtmlReporter uses the following configuration parameters:

The bug report is generated when the test run was finished or the JVM shuts down. Since most test runners spawn a separate JVM for their tests, this is a workable solution. You can however also force a generation of the test report anytime during execution by calling the static snapshot() method of BugUnit.

Automatic JIRA Synchronization

If bugunit.reporter.classes contains the jira.JiraReporter entry (which resolves to com.codestreet.bugunit.jira.JiraReporter), bug states are synchronized with a  JIRA bugtracking system. If all tests for a bug succeed, the bug is set to 'resolved'. If at least one test fails, the bug is set to 'open'.

BugUnit's JiraReporter uses the SOAP protocol to access a JIRA system. The following jar files are necessary for JIRA access only:

From Apache Axis:

  1. axis: axis-1.3.jar
  2. axis: axis-jaxrpc-1.3.jar
  3. axis: axis-wsdl4j-1.5.1.jar
  4. axis: axis-saaj-1.3.jar
  5. commons-logging: commons-logging-1.1.jar
  6. commons-discovery: commons-discovery-0.2.jar

From Sun:

  1. javamail: javamail-1.3.2.jar
  2. activation: activation-1.0.2.jar

For convenience, all of these jars are included in the <BugUnit>/lib directory.

The JiraReporter is configured through the following parameters in bugunit.properties:

Minimal Retest

AllFailingTestCases gives access to a JUnit TestSuite that contains only the previously failing bug test cases. AllFailingTestMethods gives access to a JUnit TestSuite that contains only the previously failing bug test methods.

After running thousands of tests and after fixing some issues, it is nice to be able to re-test only the test cases that previously failed. To re-test, simply run AllFailingTestCases or AllFailingTestMethods as a JUnit test or implement your own test case that uses the suite() method of these classes. The FailingMethodsReTest.java example shows how to use this approach.

Process Utilities

The classes ProcessUtils and ProcessInstance support starting, monitoring and stopping system processes on Unix or Windows in an OS-unspecific way. The current memory consumption is accessible and can be used to find memory leaks of external processes from within JUnit. See the ProcessUtilTest.java example for some base usage.
The following parameters configure the process utilities:

File Utilites

FileUtils provides some basic file utilities to recursively generate or remove directories. To prevent from accidental removal of directories, recursive removal requires the test directory to contain the String 'test'.

Extended JUnit Assertions

AssertExt extends JUnit assertions with some useful methods. Better error messages for array and String comparisons and some useful 'InRange' and 'StringContains' methods are provided. To easily debug String comparison problems, AssertExt.assertEquals(String,String) automatically prints a verbose error message to standard error:
  ERROR: name incorrect: String is:        'something'
  ERROR: name incorrect: String expected:  'pete'
  ERROR: name incorrect:        not equal at^ (index 0) found 's' (hex: 0x73) instead of 'p' (hex: 0x70)
The ExtendedAssertTest.java example demonstrates some of these extended assertions.

Robust Logging

The logger component used by BugUnit is BugUnitLogger. This class will remember standard out even when started from within Maven. It provides a method to send logging to standard out and standard error of the original main thread (circumventing Maven's modified output streams, if needed).

To debug the operation of BugUnit, enable the debugging roles through the bugunit.debug parameter. To enable all roles, set it to: "BugUnit,Monitor,Tracker,File,Process,Runner,Jira"

Method Counting

Method counting is enabled by sub-classing your test case from CountingTestCase. If you inherit from BugTestCase, this feature is enabled automatically.

If this feature is enabled, each test method knows its index in the list of executed test cases or test run or if it is the very first or very last method of a TestCase or test run. The CountingTest.java example demonstrates how methods can access their relative location inside a JUnit test case or test run.

Deadlock Detection

Deadlock detection is enabled by sub-classing your test case from MonitoredTestCase.

Deadlocking threads that were spawned from a test method are automatically detected and terminated. The maximum execution time (in seconds) for a test method is defined through the bugunit.monitor.testcase.maxtime parameter. Once a method runs longer than this interval, all threads matching the regular expression pattern of bugunit.monitor.interrupt.threads will be terminated.

This feature is very useful to run thousands of tests robustly through a Maven test runner without the risk of a hangup during the execution. The DeadlockingTest.java example file demonstrates this deadlock detection.

The bugunit.monitor.memstatfile configuration option defines the name of a memory statistics file that shows all threads before and after executing a monitored test case. If a deadlock was detected and threads had to be terminated, detailed information is shown in this file.

If your test runner is Maven, you should configure Maven to spawn a separate JVM for ever test case and set bugunit.monitor.interrupt.jvm="${maven.junit.fork}". This will ensure that also the spawned JVM is terminated in case maven.junit.fork was true.