Groovy Closures Make Unit Testing With "Soft Asserts" Simple

| Comments

A recent blog post, Cedric Beust asks about how to cleanly implement “soft asserts”. Soft asserts are test assertions that don’t “fail fast”. Instead, all of the assertion failures in the test method are collected and reported at the end of the test.

So far, the proposals in the comments look fairly clunky to me and include defining whole sets of new methods like “assertEqualsButContinue(“foo”, “foo”)“, chaning assertions together jQuery-style, or using lists to hold all of our assertions.

Wouldn’t it be a lot nicer if we could continue to use the same methods that we’re already using?

In groovy, enabling soft assertions is easy with a little closure magic.

Closures have a property called the “resolveStrategy” that determines where the code inside the closure should look to resolve method names. The default resolve strategy is OWNER_FIRST.

Closure closure = { println "foo" }
assert closure.getResolveStrategy() == Closure.OWNER_FIRST

OWNER_FIRST means that if something isn’t defined inside the closure (local scope), it should check the owner of the closure to see if it can resolve it. The owner of the closure is the declaring class or another closure that contains our closure.

Closures also have a property called the “delegate”. By default, the delegate is equal to the owner, but we have the ability to change the delegate to something else. We can also change the resolveStrategy for the closure so when something isn’t defined, the delegate is the first place that is checked.

class Original {
   def closure = { return value() }   
   def value = { return "value from owner" }

class SomethingElse {
    def value = { return "value from delegate" }

def obj = new Original()

assert "value from owner" == obj.closure()

// the owner is the enclosing class, and its the same as the delegate by default
assert obj.closure.owner.class == Original
assert obj.closure.owner == obj.closure.delegate

// but we can change the closure's delegate to something else if we want
obj.closure.delegate = new SomethingElse()

// by default, the closure looks at the owner before the delegate, value is unchanged
assert "value from owner" == obj.closure() 
assert obj.closure.getResolveStrategy() == Closure.OWNER_FIRST

// changing the closure's resolve strategy switches who gets to handle value()
obj.closure.resolveStrategy = Closure.DELEGATE_FIRST
assert "value from delegate" == obj.closure()

Groovy also has a special metaClass method called invokeMethod that allows you to manipulate any method calls that are invoked on that metaClass.

Using these two things in conjunction, we can write a small class that lets us collect JUnit assertion errors thrown by any method starting with “assert”.

class SoftAsserts {
    def oldDelegate
    def failedAssertions = []
    static softAsserts(closure) {
        new SoftAsserts().bundleAsserts(closure)
    private bundleAsserts = { closure ->
        closure.resolveStrategy = Closure.DELEGATE_ONLY
        oldDelegate = closure.delegate
        closure.delegate = this
        if (failedAssertions) {
            throw new Exception("${failedAssertions.size()} failed assertions found:\n${failedAssertions.message.join('\n')}")
    def invokeMethod(String name, args) {
        if (name.startsWith("assert")) {
            try {
                return oldDelegate.invokeMethod(name, args)
            } catch (junit.framework.AssertionFailedError e) {
                failedAssertions << e
        } else {
            return oldDelegate.invokeMethod(name, args)         

This test class does a static import of the SoftAsserts.softAsserts closure so it can use it in a natural way.

import grails.test.*
import static SoftAsserts.softAsserts

class BookTests extends GroovyTestCase {
    void testBookTrueAssertsPasses() {
        def book = [title: "Infinite Jest"]
        // all 4 assertions pass and we are able to refer to local variables 
        // like book and local methods like trueMethod() 
        softAsserts { 
            assertTrue true
            assertTrue trueMethod()
            assertNotNull book
            assertEquals "Infinite Jest", book.title

    void testBookFalseAssertsCollectsFailures() {
        def book = [title: "Infinite Jest"]

        // all 4 assertions throw errors, but all are collected before 
        // an exception is actually thrown
        softAsserts { 
            assertTrue "False is not true", false
            assertTrue "falseMethod() is not true", falseMethod()
            assertNull "Book is not null", book
            assertEquals "House of Leaves", book.title
    def trueMethod() { true }   
    def falseMethod() { false }

The first test shows how a passing test can successfully refer to other variables and methods defined inside the test class. The second test fails, but it doesn’t fail till all 4 of the assertions are tried and collected. It then throws an error with the assertion failure count and the individual error messages:

Testcase: testBookFalseAssertsCollectsFailures took 0.371 sec
    Caused an ERROR
4 failed assertions found:
False is not true
falseMethod() is not true
Book is not null
expected:<[House of Leaves]> but was:<[Infinite Jest]>

It might be useful to also print out the individual stack traces for each assertion error, but this is left as an exercise for the reader.

In most situations, I like the fail fast behavior and limiting tests to exercising one feature at a time, but there have been a few test cases where something like this would’ve been nice.