I write my own custom iterators all the time and name them something descriptive. This makes the code much more readable. Rather than trying to decipher what a for loop is trying to do, we wrap up all of that iteration logic into a meaningful name and we cleanly separate that iteration from the processing that we’re doing with each element.

This kind of design is a core concept in Uncle Bob’s Clean Code, one of my favorite programming books in the last few years.

This example iterates over a collection and calls the passed in closure until we hit a value greater than 5.

def eachUntilGreaterThanFive = { collection, closure ->
    for ( value in collection ) {
        if ( value  > 5 ) break
        closure(value)
    }
}
 
def a = [1, 2, 3, 4, 5, 6, 7]
 
eachUntilGreaterThanFive(a) {
    println it
}

prints:

1
2
3
4
5

This code makes it obvious what the iterator is doing (looping till we hit a condition) as well as what will happen with each element iterated over (print it out).

For a real life example, I had a need to iterate over a list of values and where I needed both the current object as well as a peek at the next object in the list.

Doing this is Java is a bit of a pain, but groovy makes it easy to write and (hopefully) to read, we can also add it directly onto the Collection metaClass so that it’s available for all of our Collection instances:

Collection.metaClass.eachWithPeek = { closure ->
    def last = null
    delegate?.each { current ->
        if (last) closure(last, current)
        last = current
    }
    if (last) closure(last, null)
}

These test cases show that as we iterate through the collection, we can see the current item and peek at the next one (if any). If the collection is empty, we don’t execute the closure it, and if we’re at the end of the list there isn’t anything to peek at:

[].eachWithPeek { current, peek ->
    assert false // shouldn't get here, nothing to iterate through
}
 
[1].eachWithPeek { current, peek ->
    assert current == 1
    assert peek == null  // only 1 element, nothing to peek at
}
 
def results = []
[1, 2, 3, 4, 5].eachWithPeek { current, peek ->
    results << [current, peek]
}
assert results == [[1, 2], [2, 3], [3, 4], [4, 5], [5, null]]

If you’re not familiar with build-test-data, the quick summary is that it puts a build() method on all grails domain objects. Calling that method will automatically construct and save an instance of that domain object that conforms to all of the domain’s constraints. It also allows you to override the values that you want to explicitly set. It makes your tests much cleaner and less fragile as you only need to specify the values that actually matter to a particular test method instead of building a huge graph of objects just to satisfy constraints.

The plugin has been quite stable for the past 6 months or so, and has survived upgrades from grails 1.0.X through grails 1.2.1 with minimal changes. Because of this, I’ve decided to move the version from 0.2.3 to 1.0 to indicate that the plugin is stable and ready to be used. I’ve also added the LICENSE file releasing the plugin under the Apache 2.0 open source license (the same license as grails). It was always open source, but I had neglected to add the official license file in the past.

If you’re not familiar with the build-test-data plugin, the documentation on the wiki is thorough, I’ve also given a presentation on build-test-data that explains why it’s better than other existing data generation technologies.

The biggest changes for this release compared to the last are a number of bugfixes around making sure that both sides of a one-to-many relationship get populated correctly and that there isn’t a need to refresh() from the database.

Previously, if you had an Author that hasMany Books, and each Book belongsTo an Author, you’d need to refresh the Author if you tried to build a new book with an existing author:

Author eap = Author.findByName("Edgar Allan Poe")
Book b = Book.build(author: eap, title: "The Tell-Tale Heart")
 
assertEquals eap.name, b.author.name  // works, linked in OK previously
assertEquals 1, eap.books.size() // FAILED previously WORKS now, previously the book wasn't added properly to the author side of things

The previous workaround was to call eap.refresh() to reload the author from the database, or to have the user manually addToBooks(b). Both solutions were ugly and kludgy and this issue has now been fixed.

class ClosureInterrogator {
    private Map closureValueMap = [:]
 
    static Map extractValuesFromClosure(Closure closure) {
        def interrogator = new ClosureInterrogator(closure)
        return interrogator.closureValueMap
    }
 
    private ClosureInterrogator(Closure closure) {
        def oldResolveStrategy = closure.getResolveStrategy()
        def oldDelegate = closure.getDelegate()
        closure.delegate = this
        closure.resolveStrategy = Closure.DELEGATE_FIRST
 
        try {
            closure()
        } finally {        
            closure.setDelegate(oldDelegate)
            closure.setResolveStrategy(oldResolveStrategy)
        }
    }
 
    // property getter
    def propertyMissing(String name) {
        return closureValueMap[name]
    }
 
    // property setter
    def propertyMissing(String name, value) {
        closureValueMap[name] = value
    }
 
    def methodMissing(String name, args) {
        if (args.size() == 1) {
            closureValueMap[name] = args[0]
        } else {
            closureValueMap[name] = args
        }
    }
}

This technique is the basis for all groovy mocking techniques and libraries, it’s also the main technique used to create a DSL with groovy.

Here is a sample class that has a closure (“something”) that we want to extract the values from.

class SomeClass {
    static something = {
        key1 "value1"              // calls methodMissing("key1", ["value1"])
        key2("value2")             // calls methodMissing("key2", ["value2"])
        key3 = "value3"            // calls propertyMissing("key3", "value3")
        key4 "foo", "bar", "baz"   // calls methodMissing("key4", ["foo","bar","baz"])
    }
}

If we call “extractValuesFromClosure” on the “something” closure, we’ll get back a map that has all of the values we want in it.

def closureValueMap = ClosureInterrogator.extractValuesFromClosure(new SomeClass().something)
 
assert "value1" == closureValueMap."key1"  // calls propertyMissing("key1")
assert "value2" == closureValueMap."key2"  // calls propertyMissing("key2")
assert "value3" == closureValueMap."key3"  // calls propertyMissing("key3")
assert ["foo", "bar", "baz"] == closureValueMap."key4"  // calls propertyMissing("key4")

This class will work with any type of closure, such as this closure used by the grails mail plugin to send mail.

def mailClosure = {
   to "fred@g2one.com","ginger@g2one.com"
   from "john@g2one.com"
   cc "marge@g2one.com", "ed@g2one.com"
   bcc "joe@g2one.com"
   subject "Hello John"
   body 'this is some text'
}
 
def results = ClosureInterrogator.extractValuesFromClosure(mailClosure)
println results
// prints:
// [to:[fred@g2one.com, ginger@g2one.com], from:john@g2one.com, cc:[marge@g2one.com, ed@g2one.com], bcc:joe@g2one.com, subject:Hello John, body:this is some text]
 
assert results.to.every { it.endsWith("g2one.com") }

If we wanted to assert that within a test environment, we’re only sending e-mail to a specific domain, this would be one technique to assert it before we actually send the e-mail off.

ClosureInterrogator is generic enough that it could also be used on a number of Grails/GORM domain closures such as “constraints” and “mapping”.

Groovy Config DSL Overview

In Java, configuration is normally done with properties files. They’re kind of a pain because they’re inflexible, don’t allow executable code, and don’t easily provide a heirarchy.

Groovy greatly improves on Java by building in support for a simple configuration DSL using the ConfigSlurper.

To use it, just instantiate a new ConfigSlurper and give it the name of a groovy script file that you’d like it to parse.

// SampleConfig.groovy, a simple groovy script
person {
    firstName = "Ted"
    address {
        city = "Minneapolis"
    }
}
 
// TestConfiguration.groovy
def configObject = new ConfigSlurper().parse(SampleConfig)

This creates a ConfigObject, which is a subclass of a Map and can be similarly accessed using dot notation

println configObject
// prints: [person:[firstName:Ted, address:[city:Minneapolis]]]
 
assert "Minneapolis" == configObject.person.address.city

ConfigSlurper is built into groovy and is heavily used for Grails configuration. Grails is a “convention over configuration” framework, but anytime you want to modify the conventions, you’re likely going to be touching a config file. Grails plugin authors also often use config files to drive plugin behavior.

In larger projects, these config files can start to get quite large and unwieldy.

How ConfigSlurper Does It’s Thing

DSLs in groovy, like the ConfigSlurper DSL, work by intercepting calls to methods and property setters. If you look at the internals of the ConfigSlurper.parse(Script script, URL location) method (the big one at the bottom), you’ll see that it heavily messes with the metaClass of the configuration script by creating custom versions of invokeMethod and getProperty.

Intercepting Methods

When you have a config script that says:

person {
    firstName = "ted"
}

What you’re saying is call the “person” method and pass the { firstName = "ted" } closure to it.

Normally, this would blow up because you haven’t defined a “person” method. With the modified metaClass, invokeMethod instead adds “person” as a key in the config map and then runs the closure associated with person. Any subsequent method calls or properties in that closure are assigned as the value in the map to the “person” key.

Intercepting Property Assignments

Property assignment (like firstName = "ted") works a little differently. Every script has something associated with it called a binding. The binding holds script-wide variables that aren’t explicitly declared. When parsed by ConfigSlurper, calls that would normally set variables in the script’s binding are monitored and inserted into the map at the appropriate place in the stack.

So with the modified metaClass and binding the config above turns into this when slurped:

[person:[firstName:ted]]

If we had explicitly declared the name property, it wouldn’t get in the binding, and wouldn’t be part of the resulting config map.

person {
    def firstName = "ted"
}
 
// result: [person:[:]]

Modularizing Config Scripts

I got to thinking about how nice gsp files are to manage with the ability to include other gsp fragments. It would be great if we could do the same kind of thing with config files. That would let us better manage large config files. It would also let us define external config files that we pull in at run time, or config templates that we want to override just a piece or two out of. All from inside the config file itself.

Something like:

// SecurityConfig.groovy
security {
    includeScript( SecurityDefaults )
    active = true
}
 
// SecurityDefaults.groovy
username = "mysql"
password = "sekr1t"	    
url {
    port = 2112    
}

Loading SecurityConfig would have everything in SecurityDefaults and also set active = true:

def configObject = new ConfigSlurper().parse( SecurityConfig )
 
println configObject
// [security:[username:mysql, password:sekr1t, url:[port:2112], active:true]]
 
assert configObject.security.active == true
assert configObject.security.url.port == 2112

A Solution

Getting the same functionality in config files ended up being a bit more difficult a problem to solve than I first expected. A big part of that was due to some misconceptions I had about how the ConfigSlurper was working as a DSL, as well as what exactly a groovy Script was and how it differed from a regular Groovy class.

It turns out that when a groovy script is compiled, it’s contents are placed inside a concrete implementation of the abstract Script.run() method. So this:

// MyGroovyScript.groovy
person {
    firstName = "Ted"
}

Is functionally equivalent to this:

class MyGroovyScript extends Script {
    def run() {
	person {
     	    firstName = "Ted"
        }
    }	    
}

So one way to solve the problem is to create an abstract subclass of script and implement the “includeScript” method. All “includeScript” needs to do is override the metaClass and binding of the child script so that it uses the same metaClass and binding as the parent script.

public abstract class ComposedConfigScript extends Script {
    def includeScript(scriptClass) {
        def scriptInstance = scriptClass.newInstance()
        scriptInstance.metaClass = this.metaClass
        scriptInstance.binding = new ConfigBinding(this.getBinding().callable)
        scriptInstance.&run.call()
    }
}

ConfigSlurper modifies the metaClass and binding before loading the parent class to keep track of things as the script executes, it doesn’t care who’s actually calling the methods.

This lets us do exactly what we want, the only limitation to this approach is that the parent script needs to explicitly state that it’s a class that extends ComposedConfigScript and implements the run() method:

// SecurityConfig.groovy
class SecurityConfig extends ComposedConfigScript {	        
    def run() { // normal contents of a config file go in here
 
        security {
            includeScript( SecurityDefaults )
            active = true
        }
 
    }
}
 
// SecurityDefaults.groovy
username = "mysql"
password = "sekr1t"	    
url {
    host = "example.com"
    port = 2112    
}
 
// OtherClass.groovy
def configObject = new ConfigSlurper().parse( SecurityConfig )
 
println configObject
// prints: [security:[username:mysql, password:sekr1t, url:[host:example.com, port:2112], active:true]]

Using this technique lets us modularize our config files. This helps us both better organize things as well as decouple pieces of information that don’t necessarily belong together (like log4j info and website url information).

I had tried a number of other techniques that would let me use includeScript without declaring an explict class (like using a DelegatingMetaClass for the Script class and static import) but wasn’t able to get them working to my satisfaction. Those solutions are great for most cases, but break down with all the meta-magic that’s already happening with the ConfigSlurper.

Another option that could be considered, would be to patch the ConfigSlurper implementation to directly allow includeScript functionality. I’d be happy to submit a patch that does this if people are interested in this functionality.

Lots of good questions from the people in attendance. Thanks to everyone for showing up.

Here’s a version of it on slideshow:

I also released a new bugfix version of build-test-data earlier today (0.2.2). Mostly fixed bugs, but there’s also a new config setting that lets you disable the plugin in specified environments (suggested by Scott Vlaminck @ refactr.

So far, the proposals in the comments look fairly clunky to me and include defining whole sets of new methods like “assertEqualsButContinue(“foo”, “foo”)”, chaning assertions together jQuery-style, or using lists to hold all of our assertions.

Wouldn’t it be a lot nicer if we could continue to use the same methods that we’re already using?

In groovy, enabling soft assertions is easy with a little closure magic.

Closures have a property called the “resolveStrategy” that determines where the code inside the closure should look to resolve method names. The default resolve strategy is OWNER_FIRST.

Closure closure = { println "foo" }
assert closure.getResolveStrategy() == Closure.OWNER_FIRST

OWNER_FIRST means that if something isn’t defined inside the closure (local scope), it should check the owner of the closure to see if it can resolve it. The owner of the closure is the declaring class or another closure that contains our closure.

Closures also have a property called the “delegate”. By default, the delegate is equal to the owner, but we have the ability to change the delegate to something else. We can also change the resolveStrategy for the closure so when something isn’t defined, the delegate is the first place that is checked.

class Original {
   def closure = { return value() }   
 
   def value = { return "value from owner" }
}
 
class SomethingElse {
    def value = { return "value from delegate" }
}
 
def obj = new Original()
 
assert "value from owner" == obj.closure()
 
// the owner is the enclosing class, and its the same as the delegate by default
assert obj.closure.owner.class == Original
assert obj.closure.owner == obj.closure.delegate
 
// but we can change the closure's delegate to something else if we want
obj.closure.delegate = new SomethingElse()
 
// by default, the closure looks at the owner before the delegate, value is unchanged
assert "value from owner" == obj.closure() 
assert obj.closure.getResolveStrategy() == Closure.OWNER_FIRST
 
// changing the closure's resolve strategy switches who gets to handle value()
obj.closure.resolveStrategy = Closure.DELEGATE_FIRST
assert "value from delegate" == obj.closure()

Groovy also has a special metaClass method called invokeMethod that allows you to manipulate any method calls that are invoked on that metaClass.

Using these two things in conjunction, we can write a small class that lets us collect JUnit assertion errors thrown by any method starting with “assert”.

class SoftAsserts {
    def oldDelegate
    def failedAssertions = []
 
    static softAsserts(closure) {
        new SoftAsserts().bundleAsserts(closure)
    }
 
    private bundleAsserts = { closure ->
        closure.resolveStrategy = Closure.DELEGATE_ONLY
        oldDelegate = closure.delegate
        closure.delegate = this
        closure()   
        if (failedAssertions) {
            throw new Exception("${failedAssertions.size()} failed assertions found:\n${failedAssertions.message.join('\n')}")
        }
    }
 
    def invokeMethod(String name, args) {
        if (name.startsWith("assert")) {
            try {
                return oldDelegate.invokeMethod(name, args)
            } catch (junit.framework.AssertionFailedError e) {
                failedAssertions << e
            }
        } else {
            return oldDelegate.invokeMethod(name, args)         
        }
    }
}

This test class does a static import of the SoftAsserts.softAsserts closure so it can use it in a natural way.

import grails.test.*
import static SoftAsserts.softAsserts
 
class BookTests extends GroovyTestCase {
    void testBookTrueAssertsPasses() {
        def book = [title: "Infinite Jest"]
 
        // all 4 assertions pass and we are able to refer to local variables 
        // like book and local methods like trueMethod() 
        softAsserts { 
            assertTrue true
            assertTrue trueMethod()
            assertNotNull book
            assertEquals "Infinite Jest", book.title
        }
    }
 
    void testBookFalseAssertsCollectsFailures() {
        def book = [title: "Infinite Jest"]
 
        // all 4 assertions throw errors, but all are collected before 
        // an exception is actually thrown
        softAsserts { 
            assertTrue "False is not true", false
            assertTrue "falseMethod() is not true", falseMethod()
            assertNull "Book is not null", book
            assertEquals "House of Leaves", book.title
        }   
    }
 
    def trueMethod() { true }   
 
    def falseMethod() { false }
}

The first test shows how a passing test can successfully refer to other variables and methods defined inside the test class. The second test fails, but it doesn’t fail till all 4 of the assertions are tried and collected. It then throws an error with the assertion failure count and the individual error messages:

Testcase: testBookFalseAssertsCollectsFailures took 0.371 sec
	Caused an ERROR
4 failed assertions found:
False is not true
falseMethod() is not true
Book is not null
expected:<[House of Leaves]> but was:<[Infinite Jest]>

It might be useful to also print out the individual stack traces for each assertion error, but this is left as an exercise for the reader.

In most situations, I like the fail fast behavior and limiting tests to exercising one feature at a time, but there have been a few test cases where something like this would’ve been nice.

Sometimes, creating a subclass isn’t feasible, often because you don’t control all of the places where that class gets used.

Calling super() without the subclass

All you need to do is to get a reference to the old method. Getting methods from the metaClass is easy, if there’s only one method signature for that method, just use the “&” operator:

def oldToOctalString = Long.metaClass.&toOctalString

If there are multiple method signatures for a method, you’ll need to use the getMetaMethod method and pass in the signature of the method you want. This is probably the safest way to ensure that you’re getting the method that you think you’re getting.

println Long.metaClass.methods.findAll { it.name == "compareTo" }.join("\n")
// prints:
// public int java.lang.Long.compareTo(java.lang.Long)
// public int java.lang.Long.compareTo(java.lang.Object)
 
def oldCompareTo = Long.metaClass.getMetaMethod("compareTo", [java.lang.Long] as Class[])

Now that you know how to get a reference to the method you want, you can redefine the method, while still letting the old method’s implementation do all the heavy lifting for you.

Here, we redefine the “plus” method on Integer to always work with absolute values, without actually getting our hands dirty in all that messy arithmetic:

def oldPlus = Integer.metaClass.getMetaMethod("plus", [Integer] as Class[])
 
Integer.metaClass.plus = { Integer n ->
    return oldPlus.invoke( Math.abs(delegate), Math.abs(n) )
}
 
assert 4 == 2 + 2
assert 4 == -2 + -2

If you don’t save the old method (or completely re-implement the logic for plus), you’ll end up in an infinite loop and throw a java.lang.StackOverflowError:

Integer.metaClass.plus = { Integer n ->
    return Math.abs(delegate) + Math.abs(n)
}
 
// DOESN'T WORK THROWS java.lang.StackOverflowError
assert 4 == -2 + -2	// FAILS!!!

This same kind of thing can be done with AOP, but that often feels more heavy than a little metaClass manipulation.

"foo".metaClass.methods*.name

Result:

["equals", "getClass", "hashCode", "notify", "notifyAll", "toString", "wait", "wait", 
"wait", "charAt", "codePointAt", "codePointBefore", "codePointCount", "compareTo", 
"compareTo", "compareToIgnoreCase", "concat", "contains", "contentEquals", 
"contentEquals", "copyValueOf", "copyValueOf", "endsWith", "equals", 
"equalsIgnoreCase", "format", "format", "getBytes", "getBytes",
 "getBytes", "getChars", "hashCode", "indexOf", "indexOf", "indexOf", 
"indexOf", "intern", "lastIndexOf", "lastIndexOf", "lastIndexOf", "lastIndexOf", 
"length", "matches", "offsetByCodePoints", "regionMatches", "regionMatches", 
"replace", "replace", "replaceAll", "replaceFirst", "split", "split", "startsWith",
 "startsWith", "subSequence", "substring", "substring", "toCharArray", 
"toLowerCase", "toLowerCase", "toString", "toUpperCase", "toUpperCase", 
"trim", "valueOf", "valueOf", "valueOf", "valueOf", "valueOf", "valueOf", 
"valueOf", "valueOf", "valueOf"]

There’s a lot of duplication in there, and the order is random, so I’ll often fix that like this:

"foo".metaClass.methods*.name.sort().unique()

Result:

["charAt", "codePointAt", "codePointBefore", "codePointCount", "compareTo",
 "compareToIgnoreCase", "concat", "contains", "contentEquals", "copyValueOf", 
"endsWith", "equals", "equalsIgnoreCase", "format", "getBytes", "getChars",
 "getClass", "hashCode", "indexOf", "intern", "lastIndexOf", "length", "matches", 
"notify", "notifyAll", "offsetByCodePoints", "regionMatches", "replace", "replaceAll",
 "replaceFirst", "split", "startsWith", "subSequence", "substring", "toCharArray", 
"toLowerCase", "toString", "toUpperCase", "trim", "valueOf", "wait"]

If you haven’t seen that “*” in metaClass.methods*.name, that’s known as the Spread Operator and it’s essentially the same thing as saying

"foo".metaClass.methods.collect { method -> method.name }.sort().unique()

If you’re using Grails and want to know what methods are on your Author class, looking at the groovy class file will give you a clue if you know the patterns, but it can be easy to forget what the exact convention is:

class Author {
	String name
	def hasMany = [books: Book]
}

Just start up a grails console and ask an Author what it can do:

println new Author().metaClass.methods*.name.sort().unique()

This will result in:

["$static_methodMissing", "<init>", "asType", "clearErrors", "create", 
"decodeBase64", "decodeHTML", "decodeJSON", "decodeJavaScript",
"decodeURL", "decodeXML", "encodeAsBase64", "encodeAsHTML",
 "encodeAsJSON", "encodeAsJavaScript", "encodeAsURL", "encodeAsXML", 
"equals", "findAll", "getBooks", "getClass", "getConstraints", "getErrors", 
"getHasMany", "getId", "getLog", "getMetaClass", "getName", "getProperties", 
"getProperty", "getVersion", "hasErrors", "hashCode", "ident", "invokeMethod", 
"methodMissing", "notify", "notifyAll", "setBooks", "setErrors", "setHasMany",
"setId", "setMetaClass", "setName", "setProperties", "setProperty", 
"setVersion", "toString", "validate", "wait"]

In there, you can see the methods that were decorated on the Author object by grails including encodeAs and decode as well as getters and setters for explicit and implicit variables. One limitation of this method is that some methods are only added by grails dynamically once the method is executed. Notice how there isn’t an addToBooks method in the list? That’s because the method really isn’t there right now. It gets added by grails as part of a methodMissing call. We can force grails to decorate our class by calling one of the dynamic methods (like count()):

def a = new Author()
def origMethods = a.metaClass.methods*.name.sort().unique()
// call one of the dynamic methods so grails fully decorates the metaClass
a.count() 
def newMethods = a.metaClass.methods*.name.sort().unique()
println newMethods - origMethods  // what methods got added?

Subtracting the original methods from newly decorated method list will give you this set of dynamic methods that grails adds when you ask for a dynamic method.

["addToBooks", "count", "createCriteria", "delete", "discard", "executeQuery", 
"executeUpdate", "exists", "find", "findAllWhere", "findWhere", "get", "getAll", 
"list", "lock", "merge", "refresh", "removeFromBooks", "save", "withCriteria",
 "withTransaction"]

If you see a method that you want to use, but can’t remember the signature, again, all you have to do is ask:

println new Author().metaClass.methods.findAll {it.name == "withCriteria"}*.parameterTypes

Result showing the 2 method signatures for withCriteria:

[[interface java.util.Map, class groovy.lang.Closure], [class groovy.lang.Closure]]

There is also a properties method on the metaClass that you can query in a similar manner:

println new Author().metaClass.properties*.name.sort().unique()

Result:

["books", "class", "constraints", "errors", "hasMany", "id", "log", "metaClass", 
"name", "properties", "version"]

Knowing about the information that the metaClass puts at your fingertips makes it much easier to poke around in the grails console, or with a simple script. You can learn a lot of hidden features about a class if you just know how to ask.