BugAssert.assertEqualsBug("TST-6745","name incorrect","pete",name);and an HTML bug report will be generated that cross-references bugs with entries in your bug tracker:
With a JIRA bug tracker, automatic state synchronization is supported. If all tests for a bug succeed, the bug is set to 'resolved'. If at least one test fails, the bug is set to 'open'. BugUnit facilitates continuous integration as it prepares Junit tests to integrate better with continuous build systems such as Maven, Cruise Control and your bug tracker.
The generated Bug Report shows for every bug:
Besides this 'bug-awareness', BugUnit includes the following optional utilities to facilitate functional testing:
To run the examples, follow these steps:
A good practice (using a test-first approach) is to write a new JUnit test exposing this bug:
public class ExampleJUnitTest extends TestCase { public void testValueNotRight() { ... get the value through the method to test ... assertEquals("testing TST-6745, value incorrect.","pete",value); } }In an ideal world, each bug is fixed before the next release of the software.
In reality, however, some bugs are of lower priority and don't need to be fixed immediately. Especially, with a continuous build process, the next release could only be days away. For some bugs you could have made the decision that a fix to these bugs will only come in some future release. On the other hand, continuous build system (such as Cruise Control) or your build manager (such as Maven) will require that all tests succeed before even continuing with additional packaging steps. With standard JUnit, a system is considered 'broken' as soon as one test fails.
What to do ?
One alternative would be to comment out thee bug-assertions or changing assertEquals() to assertNotEquals().
But will you remember to change it back, once the bug is fixed ? This is good
test code. Why artificially change it ?
Or you could simply get used to the fact that some
of your hundreds or thousands of tests fail. But then, will you recognize that
you just discovered a new bug when your number of JUnit failures climbed from 47
to 48 ?
With JUnit as used above, there is also no central place to check which bugs are currently 'open' and which are 'closed'. Since bugs are more coarse grained than JUnit failures (a single bug in the system can cause many JUnit assertions to fail), JUnit alone could never solve this problem.
public class ExampleTest extends BugTestCase { public void testNameNotRight() { ... assertEqualsBug("TST-6745","name incorrect","pete",name); } }Instead of inheriting your test case from BugTestCase, you could also use the static methods of BugAssert as shown below. This approach is less intrusive as it does not require to change your existing TestCase type hierarchy.
public class ExampleStaticTest extends TestCase { public void testNameNotRight() { ... BugAssert.assertEqualsBug("TST-6745","name incorrect","pete",name); } }Running such a BugUnit-enabled test will generate an issues.xml file that contains all stack-traces of failing and succeeding bug tests. At the end of a test run, the data model of issues.xml is passed to a set of pluggable Reporters. These reporters generate a Text or HTML Bug Report or can automatically update your bugtracking system (e.g. JIRA).
When running a large number of tests automatically, it can further be beneficial
to ignore open bugs. To let JUnit ignore all open bugs, set the
bugunit.openbugs.throw parameter to false
.
In this case, open bugs will not throw a JUnit AssertionError and Cruise Control
or Maven will continue assuming a 'sane' system. The JUnit report will then only
show newly discovered problems. Known bugs are shown in BugUnit's
Bug Report.
Once a bug is fixed, you might want to be notified, also in JUnit, if it breaks again.
To achieve this, you can use the assert..BugFix()
instead of
the assert..Bug()
methods.
These BugFix assertions will throw JUnit errors even if bugunit.openbugs.throw
was set to false
.
public class ExampleBugUnitTest extends BugTestCase { public void testNameNotRight() { ... BugAssert.assertEqualsBugFix("TST-6745","name incorrect","pete",name); } }
Tracking of bugs can be accumulative or not.
Accumulative bug tracking (bugunit.tracker.accumulate=true
)
is useful when you iteratively explore your test cases. Initially you should remove issues.xml
and then run a group of test cases, for example in your IDE. Then run another group of tests.
With accumulative testing, BugUnit will add new bugs to the previously found
ones in issues.xml. The current bug report will contain all bugs since
issues.xml was last deleted. This setting is useful for manual and iterative
exploration of tests.
With bugunit.tracker.accumulate=false
, BugUnit will overwrite issues.xml
with every test run. This setting is useful when you are running all tests in
one go, for example through Maven, and want to start with a fresh issues.xml
file.
To disable the tracking of bug tests, set bugunit.tracker.trackbugs to false
.
Issues are no longer stored to issues.xml and your test software will behave as
if you were using standard Junit assertions. The bug ids in the bug assertions
are ignored completely. This setting is useful to temporarily disable BugUnit's
'bug-awareness'.
To write a new reporter, extend the Reporter interface and add its classname to the bugunit.reporter.classes parameter. bugunit.reporter.classes contains a comma-separated list of reporter class which are invoked in the specified order.
The HtmlReporter uses the following configuration parameters:
If bugunit.reporter.classes contains the jira.JiraReporter entry (which resolves to com.codestreet.bugunit.jira.JiraReporter), bug states are synchronized with a JIRA bugtracking system. If all tests for a bug succeed, the bug is set to 'resolved'. If at least one test fails, the bug is set to 'open'.
BugUnit's JiraReporter uses the SOAP protocol to access a JIRA system. The following jar files are necessary for JIRA access only:
From Apache Axis:
From Sun:
For convenience, all of these jars are included in the <BugUnit>/lib directory.
The JiraReporter is configured through the following parameters in bugunit.properties:
AllFailingTestCases gives access to a JUnit TestSuite that contains only the previously failing bug test cases. AllFailingTestMethods gives access to a JUnit TestSuite that contains only the previously failing bug test methods.
After running thousands of tests and after fixing some issues, it is nice to be able to re-test only the test cases that previously failed. To re-test, simply run AllFailingTestCases or AllFailingTestMethods as a JUnit test or implement your own test case that uses the suite() method of these classes. The FailingMethodsReTest.java example shows how to use this approach.
AssertExt.assertEquals(String,String)
automatically prints a verbose
error message to standard error:
ERROR: name incorrect: String is: 'something' ERROR: name incorrect: String expected: 'pete' ERROR: name incorrect: not equal at^ (index 0) found 's' (hex: 0x73) instead of 'p' (hex: 0x70)The ExtendedAssertTest.java example demonstrates some of these extended assertions.
To debug the operation of BugUnit, enable the debugging roles through the bugunit.debug parameter. To enable all roles, set it to: "BugUnit,Monitor,Tracker,File,Process,Runner,Jira"
If this feature is enabled, each test method knows its index in the list of executed test cases or test run or if it is the very first or very last method of a TestCase or test run. The CountingTest.java example demonstrates how methods can access their relative location inside a JUnit test case or test run.
Deadlocking threads that were spawned from a test method are automatically detected and terminated. The maximum execution time (in seconds) for a test method is defined through the bugunit.monitor.testcase.maxtime parameter. Once a method runs longer than this interval, all threads matching the regular expression pattern of bugunit.monitor.interrupt.threads will be terminated.
This feature is very useful to run thousands of tests robustly through a Maven test runner without the risk of a hangup during the execution. The DeadlockingTest.java example file demonstrates this deadlock detection.
The bugunit.monitor.memstatfile configuration option defines the name of a memory statistics file that shows all threads before and after executing a monitored test case. If a deadlock was detected and threads had to be terminated, detailed information is shown in this file.
If your test runner is Maven, you should configure Maven to spawn a separate JVM for ever test case and set bugunit.monitor.interrupt.jvm="${maven.junit.fork}". This will ensure that also the spawned JVM is terminated in case maven.junit.fork was true.