I write my own custom iterators all the time and name them something descriptive. This makes the code much more readable. Rather than trying to decipher what a for loop is trying to do, we wrap up all of that iteration logic into a meaningful name and we cleanly separate that iteration from the processing that we’re doing with each element.

This kind of design is a core concept in Uncle Bob’s Clean Code, one of my favorite programming books in the last few years.

This example iterates over a collection and calls the passed in closure until we hit a value greater than 5.

def eachUntilGreaterThanFive = { collection, closure ->
    for ( value in collection ) {
        if ( value  > 5 ) break
        closure(value)
    }
}
 
def a = [1, 2, 3, 4, 5, 6, 7]
 
eachUntilGreaterThanFive(a) {
    println it
}

prints:

1
2
3
4
5

This code makes it obvious what the iterator is doing (looping till we hit a condition) as well as what will happen with each element iterated over (print it out).

For a real life example, I had a need to iterate over a list of values and where I needed both the current object as well as a peek at the next object in the list.

Doing this is Java is a bit of a pain, but groovy makes it easy to write and (hopefully) to read, we can also add it directly onto the Collection metaClass so that it’s available for all of our Collection instances:

Collection.metaClass.eachWithPeek = { closure ->
    def last = null
    delegate?.each { current ->
        if (last) closure(last, current)
        last = current
    }
    if (last) closure(last, null)
}

These test cases show that as we iterate through the collection, we can see the current item and peek at the next one (if any). If the collection is empty, we don’t execute the closure it, and if we’re at the end of the list there isn’t anything to peek at:

[].eachWithPeek { current, peek ->
    assert false // shouldn't get here, nothing to iterate through
}
 
[1].eachWithPeek { current, peek ->
    assert current == 1
    assert peek == null  // only 1 element, nothing to peek at
}
 
def results = []
[1, 2, 3, 4, 5].eachWithPeek { current, peek ->
    results << [current, peek]
}
assert results == [[1, 2], [2, 3], [3, 4], [4, 5], [5, null]]

If you’re not familiar with build-test-data, the quick summary is that it puts a build() method on all grails domain objects. Calling that method will automatically construct and save an instance of that domain object that conforms to all of the domain’s constraints. It also allows you to override the values that you want to explicitly set. It makes your tests much cleaner and less fragile as you only need to specify the values that actually matter to a particular test method instead of building a huge graph of objects just to satisfy constraints.

The plugin has been quite stable for the past 6 months or so, and has survived upgrades from grails 1.0.X through grails 1.2.1 with minimal changes. Because of this, I’ve decided to move the version from 0.2.3 to 1.0 to indicate that the plugin is stable and ready to be used. I’ve also added the LICENSE file releasing the plugin under the Apache 2.0 open source license (the same license as grails). It was always open source, but I had neglected to add the official license file in the past.

If you’re not familiar with the build-test-data plugin, the documentation on the wiki is thorough, I’ve also given a presentation on build-test-data that explains why it’s better than other existing data generation technologies.

The biggest changes for this release compared to the last are a number of bugfixes around making sure that both sides of a one-to-many relationship get populated correctly and that there isn’t a need to refresh() from the database.

Previously, if you had an Author that hasMany Books, and each Book belongsTo an Author, you’d need to refresh the Author if you tried to build a new book with an existing author:

Author eap = Author.findByName("Edgar Allan Poe")
Book b = Book.build(author: eap, title: "The Tell-Tale Heart")
 
assertEquals eap.name, b.author.name  // works, linked in OK previously
assertEquals 1, eap.books.size() // FAILED previously WORKS now, previously the book wasn't added properly to the author side of things

The previous workaround was to call eap.refresh() to reload the author from the database, or to have the user manually addToBooks(b). Both solutions were ugly and kludgy and this issue has now been fixed.

class ClosureInterrogator {
    private Map closureValueMap = [:]
 
    static Map extractValuesFromClosure(Closure closure) {
        def interrogator = new ClosureInterrogator(closure)
        return interrogator.closureValueMap
    }
 
    private ClosureInterrogator(Closure closure) {
        def oldResolveStrategy = closure.getResolveStrategy()
        def oldDelegate = closure.getDelegate()
        closure.delegate = this
        closure.resolveStrategy = Closure.DELEGATE_FIRST
 
        try {
            closure()
        } finally {        
            closure.setDelegate(oldDelegate)
            closure.setResolveStrategy(oldResolveStrategy)
        }
    }
 
    // property getter
    def propertyMissing(String name) {
        return closureValueMap[name]
    }
 
    // property setter
    def propertyMissing(String name, value) {
        closureValueMap[name] = value
    }
 
    def methodMissing(String name, args) {
        if (args.size() == 1) {
            closureValueMap[name] = args[0]
        } else {
            closureValueMap[name] = args
        }
    }
}

This technique is the basis for all groovy mocking techniques and libraries, it’s also the main technique used to create a DSL with groovy.

Here is a sample class that has a closure (“something”) that we want to extract the values from.

class SomeClass {
    static something = {
        key1 "value1"              // calls methodMissing("key1", ["value1"])
        key2("value2")             // calls methodMissing("key2", ["value2"])
        key3 = "value3"            // calls propertyMissing("key3", "value3")
        key4 "foo", "bar", "baz"   // calls methodMissing("key4", ["foo","bar","baz"])
    }
}

If we call “extractValuesFromClosure” on the “something” closure, we’ll get back a map that has all of the values we want in it.

def closureValueMap = ClosureInterrogator.extractValuesFromClosure(new SomeClass().something)
 
assert "value1" == closureValueMap."key1"  // calls propertyMissing("key1")
assert "value2" == closureValueMap."key2"  // calls propertyMissing("key2")
assert "value3" == closureValueMap."key3"  // calls propertyMissing("key3")
assert ["foo", "bar", "baz"] == closureValueMap."key4"  // calls propertyMissing("key4")

This class will work with any type of closure, such as this closure used by the grails mail plugin to send mail.

def mailClosure = {
   to "fred@g2one.com","ginger@g2one.com"
   from "john@g2one.com"
   cc "marge@g2one.com", "ed@g2one.com"
   bcc "joe@g2one.com"
   subject "Hello John"
   body 'this is some text'
}
 
def results = ClosureInterrogator.extractValuesFromClosure(mailClosure)
println results
// prints:
// [to:[fred@g2one.com, ginger@g2one.com], from:john@g2one.com, cc:[marge@g2one.com, ed@g2one.com], bcc:joe@g2one.com, subject:Hello John, body:this is some text]
 
assert results.to.every { it.endsWith("g2one.com") }

If we wanted to assert that within a test environment, we’re only sending e-mail to a specific domain, this would be one technique to assert it before we actually send the e-mail off.

ClosureInterrogator is generic enough that it could also be used on a number of Grails/GORM domain closures such as “constraints” and “mapping”.

I’ve made some enhancements to them in the interim that have made them even easier to use.

The most important alias is gtaf, which is short for “grails test-app” for failed tests.

It will search through your test output directory and look for any tests that failed. If it finds any, it will rerun only those tests. Otherwise, it will rerun all tests. That makes it easy to just use gtaf all the time. If any tests fail, it will open them up using Console.app.

If you’re not on OSX, or would like to use something else to view the failed test logs, just modify the testlog alias to do something different.

There are alternative versions of it to only run failed integration tests (gtaif – Grails Test App Integration Failed), failed unit tests (gtauf – Grails Test App Unit Failed) or attach a debugger to the tests as they run (gtadf – Grails Test App Debug Failed).

These aliases, plus most of the rest of my zsh setup, is available on bitbucket. If you haven’t used zsh before and are still using bash, I suggest switching over for the numerous benefits that it gives you.

Here’s the section of zshrc_general that has the grails testing aliases, just add this to your .zshrc/.bashrc and make sure to uncomment the appropriate GRAILS_TEST_LOG_DIRECTORY export if you’re not running grails 1.2 yet:

# grails > 1.2
export GRAILS_TEST_LOG_DIRECTORY=target/test-reports
 
# grails < 1.2
# export GRAILS_TEST_LOG_DIRECTORY=test/reports
 
# after grails-test if there were ERROR messages, you can open those logs with the Console.app using this
alias testlog='for F in `grep -lE "FAILED|Caused\ an\ ERROR" $GRAILS_TEST_LOG_DIRECTORY/plain/*.txt`; do echo ">>> opening" $F; open -a Console $F; done;'
 
# grails-debug-suspend doesn't exist by default, it's a copy of grails-debug with the suspend flag changed to "y" so that
# we can attach a remote debugger before it proceeds past startup
 
# aliases where you can optionally pass in a set of tests to run (or no argument to run all tests in that group)
alias gta=grailsTestApp
alias gtad=grailsTestAppDebug
alias gtau=grailsTestAppUnit
alias gtaud=grailsTestAppUnitDebug
alias gtai=grailsTestAppIntegration
alias gtaid=grailsTestAppIntegrationDebug
 
# aliases that will rerun any failed tests (or all tests if there aren't any failed tests)
alias gtaf=grailsTestAppFailed
alias gtadf=grailsTestAppDebugFailed
alias gtauf=grailsTestAppUnitFailed
alias gtaudf=grailsTestAppUnitDebugFailed
alias gtaif=grailsTestAppIntegrationFailed
alias gtaidf=grailsTestAppIntegrationDebugFailed
 
 
function grailsTestApp() { grailsTest grails "" $1 }
function grailsTestAppFailed() { grailsFailedTests grails "" }
 
function grailsTestAppDebug() { grailsTest grails-debug-suspend "" $1 }
function grailsTestAppDebugFailed() { grailsFailedTests grails-debug-suspend "" }
 
function grailsTestAppUnit() { grailsTest grails -unit $1 }
function grailsTestAppUnitFailed() { grailsFailedTests grails -unit }
 
function grailsTestAppUnitDebug() { grailsTest grails-debug-suspend -unit $1 }
function grailsTestAppUnitDebugFailed() { grailsFailedTests grails-debug-suspend -unit }
 
function grailsTestAppIntegration() { grailsTest grails -integration $1 }
function grailsTestAppIntegrationFailed() { grailsFailedTests grails -integration }
 
function grailsTestAppIntegrationDebug() { grailsTest grails-debug-suspend -integration $1 }
function grailsTestAppIntegrationDebugFailed() { grailsFailedTests grails-debug-suspend -integration }
 
function grailsFailedTests() {
	FAILED_TESTS=''
	for F in `grep -lE "FAILED|Caused\ an\ ERROR" $GRAILS_TEST_LOG_DIRECTORY/plain/*.txt`; do
		FAILED_TESTS="$FAILED_TESTS `echo $F | sed -E 's/.*TEST-(.*)Tests.txt/\1/g'`"
	done;
	echo "failed tests: $FAILED_TESTS"
	grailsTest $1 $2 $FAILED_TESTS
}
 
function grailsTest() {
	echo "Running: $1 test-app $2 $3 || testlog "
    time $1 test-app $2 $3 || testlog
}

I believe that they’ll also work fine as-is in bash, though I haven’t tested them.

I’m a big fan of storing information in markdown format as it is easy to read, easy to write, and easy to transform. A number of big websites like stackoverflow.com support the entry and display of markdown text to help people format their questions and answers without having to remember to properly format HTML.

If you’re unfamiliar with Markdown there are a number of good references on it’s use.

The grails markdown plugin is a very basic plugin. Currently, it just wraps the markdown libarary with a single TagLib that lets you easily render html from a markdown string within your gsp pages.

To install it, just type:

grails install-plugin markdown

To use it in your gsp file:

<markdown:renderHtml>
The *four* cardinal directions are
 
- North
- South
- East
- West
</markdown:renderHtml>

renders the html:

<p>The <em>four</em> cardinal directions are</p>
<ul>
    <li>North</li>
    <li>South</li>
    <li>East</li>
    <li>West</li>
</ul>

Or you can use the “text” attribute of the taglib:

<markdown:renderHtml text="${post.body}"/>

See the Grails Markdown Wiki and source for more details.

A number of interesting techniques have popped out of this that I think would be useful for others doing similar types of importing using GORM/Hibernate/MySQL.

Background

For background, the imports in our app happen a few times a week, and range in size from less than a hundred up to a few hundred thousand new records.

It is also possible for us to get updates on previous data that we’ve imported, if a record in the import matches a record in our database, we want to update the existing record rather than inserting a duplicate.

We had a desire to use a Grails service to do the importing as we could leverage our existing data model, plus, there are some special cases where the import logic can get a bit hairy and having access to all of our domain methods is nice.

If you’re doing much more frequent imports, with heavy performance requirements using millions to hundreds of millions of rows, you’ll likely want to look at another tool more specifically targeted for batch/ETL processes or writing something directly in Java. For smaller jobs, you might be surprised how far you can get with groovy and grails.

Example Naive Import Application

To demonstrate the techniques, I’ve created a small test application. This simple app only has Books. It imports 100,000 randomly generated books in a single job (importing a Library of Books). If our database already has a book with that ISBN and edition number, we want to update the title of previous book rather than creating a new book.

Here are the setup files that won’t change as we tweak our application.

The Domain class:

// domain/Book.groovy
class Book {
    String title
    String isbn
    Integer edition
}

The BookController that calls the service:

// controller/BookController.groovy
class BookController {
 
    def bookService
 
    def index = { 
        def library = new Library()
        def startTime = System.currentTimeMillis()		
        bookService.importBooksInLibrary(library)	
        render "time: ${(startTime - System.currentTimeMillis())/1000} seconds"
    }
}

And a Library class that’s the source of our books (don’t worry too much about the details of the Library class, just know that it’s the source of the fields that we’ll be importing. For each book, it simply returns a map of the field values that should be saved or updated for the Book):

class Library implements Iterator {
    def startTime = System.currentTimeMillis()
    def lastBatchStarted = startTime
    def BOOKS_IN_LIBRARY = 100000
    def currentBook = 1
    def random = new Random(12345) // random generator with defined seed
 
    Iterator iterator() { return this as Iterator }
    boolean hasNext() { currentBook <= BOOKS_IN_LIBRARY }
    void remove() { }
    def next() {
        if (! (currentBook % 100) ) printStatus()
 
        return [
            title: "Book ${currentBook++}",
            isbn: randomIsbn(), 
            edition: randomEdition()
        ]
    }
 
    def randomIsbn() {
        // one of 50,000 random isbn numbers
        return "isbn-${random.nextInt(50000)}"
    }
 
    def randomEdition() {
        // first through tenth editions
        return random.nextInt(9) + 1
    }
 
    def printStatus() {
        def batchEnded = System.currentTimeMillis()
        def seconds = (batchEnded-lastBatchStarted)/1000
        def total = (batchEnded-startTime)/1000
        println "Library Book $currentBook, last batch: ${seconds}s, total: ${total}s"
        lastBatchStarted = batchEnded
    }
}

The Library class is a proxy for whatever the source of your import info is (a .csv/excel file that you’re parsing through, a webservice, something posted to your server, etc.).

Also, to help see how we affect timing, every 100 books the Library prints out how long it took to process the last 100 books.

Here is the naive BookService that iterates through the Library that the BookController gives it. For each book it receives a map of the books values. It does a findByIsbnAndEdition() to see if the book already exists. If it does, it updates it, otherwise it will create a new book:

class BookService {
    def importBooksInLibrary(library) {		
        library.each { Map bookValueMap ->
            updateOrInsertBook(bookValueMap)
        }		
    }
 
    def updateOrInsertBook(Map bookValueMap) {
        def title = bookValueMap.title
        def isbn = bookValueMap.isbn
        def edition = bookValueMap.edition
        def existingBook = Book.findByIsbnAndEdition(isbn, edition)
 
        if (existingBook) { // just update title
            existingBook.title = title
            existingBook.save()
        } else { 
            new Book(bookValueMap).save()
        }
    }
}

When you run this code, the first thing you notice is how slow it is. The first 1000 records aren’t that bad. On my old laptop they take about 7 seconds to insert. After that things get slower and slower, books 10,000 to 11,000 take 20 seconds to insert. Importing gets progressively worse as time goes on.

Yikes! It takes 2 hours and 36 minutes to import 100k records? Over 3 minutes for the last thousand books to get imported? That can’t be right.

As it runs, if you open up your activity monitor/task manager/top application, you’ll also see that Grails is running very hot but also that MySQL’s CPU usage is slowly going up over time.

MySQL Database Performance Tweaks

Start with the obvious: Database Indexes

If you’ve done much database work, you know how much indexes can improve the speed of a query. But how do you know what queries need indexes?

In some situations, like our example above, it can be fairly obvious. We’re calling Book.findByIsbnAndEdition(). Chances are we need an index on the “book” table that includes the “isbn” and “edition” fields:

CREATE INDEX `idx_book_isbn_edition` ON `book`(`isbn`, `edition`);

What if it isn’t so obvious which queries need an index?

There are a number of techniques available to us through grails to see what kind of SQL is getting executed.

Turn on grails SQL logging

Within grails, you can turn on SQL logging by putting this in your Config.groovy’s log4j closure:

trace 'org.hibernate.SQL'

Install the p6spy plugin and JDBC SQL profiler

The p6spy grails plugin inserts a proxy between your database driver classes and Grails. Because of this, it knows the full query, including the filled in query parameters and some timing information.

This is useful by itself, but you can have the p6spy plugin log it’s messages to a log4j socket appender that talks to the JDBC SQL Profiler which will aggregate all of the information and help you decide where you’ve got slow queries and need indexes.

Mike Hugo has a great post on using the p6spy plugin with the JDBC SQL profiler.

Turn on the MySQL Slow Query Log

If you’re using a recent version of MySQL, there’s an easy, built-in way to see which queries are slow and which queries aren’t using indexes: the slow query log.

To turn it on, you simply need to add a few lines to your mysql cnf file (by default on OSX just create a file at /etc/my.cnf and it will get loaded):

[mysqld]
log-slow-queries
#Log all queries > long_query_time seconds, default 10 seconds
long_query_time=1

This will automatically create a log file in your mysql “data” directory called host_name-slow.log that contains all queries taking longer than 1 second.

You can optionally add

log-queries-not-using-indexes

And this will also log those queries that don’t have an index (but don’t take longer than 1 second to execute).

MySQL also comes with a perl script to parse through the slow log file and show you the number of times each query is slow with some additional statistics. To run it, you need to add basedir to your my.cnf file. The basedir value just holds the location of mysql’s data directory.

basedir=/usr/local/mysql

Then run the script:

mysqldumpslow

These steps should get you a long way towards figuring out which tables need indexes.

Turn on the MySQL General Query log

MySQL has an additional query log called the general query log. It logs everything that comes into MySQL and is extremely noisy. One benefit of it though is that it gives you the full query with all parameters, so it’s more useful than turning on grails SQL trace log and potentially a little quicker to set up than the p6spy plugin.

You can enable it by putting this in your my.cnf:

general_log=1

Suggested Reading for MySQL indexes

If you haven’t created many indexes before, I highly suggest reading this post on indexes on the mysql high performance blog. The executive summary is that multi-key indexes tailored to your query have a performance edge over individual key indexes on each of the fields. If you’re executing a query enough times, it might be worth creating a multi-key index like we have above.

MySQL Database Engines

If MySQL is still your performance bottleneck, or if you’re finding that everything else comes to a stop when you’re running your batch import, check which storage engine you’re using.

For grails applications, I’d prefer InnoDB (or Falcon) to the default MyISAM. MyISAM is good for read-only situations where you don’t care about enforced foreign key releationships. Switching to InnoDB has significantly increased the performance of my app in a number of situations. InnoDB also supports row level locking, whereas MyISAM will lock the entire table when doing an update or an insert.

If you haven’t explicitly done anything, chances are that all of your grails-generated tables are MyISAM. You can change the default storage engine for mysql by specifying a different engine in your my.cnf:

default-storage-engine=innodb

To see the default is for your database execute:

SHOW engines;

To see the engine for a particular table, use “show create table”:

mysql> SHOW CREATE TABLE book\G
*************************** 1. row ***************************
       TABLE: book
CREATE TABLE: CREATE TABLE `book` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `version` bigint(20) NOT NULL,
  `edition` int(11) NOT NULL,
  `isbn` varchar(255) NOT NULL,
  `title` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=84192 DEFAULT CHARSET=latin1
1 row IN SET (0.00 sec)

If you need to alter the storage engine for an existing table, you can execute the alter table SQL statement:

ALTER TABLE book ENGINE = INNODB;

Grails Performance Tweaks

Adding an index and modifying the storage engine of our example book table to InnoDB make our sample app 25-30% faster. A nice boost from where we were originally, but things are still way too slow:

Adding indexes to the database drops the database CPU utilization to almost nothing, but grails is still pegged and it still gets much slower over time.

Something is amiss.

At this point, some people might be inclined to believe the myth that “grails is slow” and go through a painful rewrite in another “faster” language/technology.

Don’t go there yet! It’s only slow if you don’t understand what’s actually going on under the covers.

After reading through a couple of very helpful posts by Burt Beckwith, and reading through the Hibernate chapter on batch processing, we have all of the information we need to speed things up significantly.

Use a Monitoring Tool to Watch Memory/CPU usage Over Time

Because the import gets significantly slower over time, even with the index applied, we know there is some sort of leak.

If you don’t have your code instrumented with timing logic already in it, it’s easy to pop open jconsole or jvisualvm (both already installed and likely in your classpath if you have Java 1.6), connect to your grails instance and watch performance/cpu utilization over time.

The Culprits: Hibernate First-Level Cache and Grails Validation Cache

I found on Burt Beckwith’s blog, that there are 2 separate leaks, one of them is in the hibernate first-level cache, the other is a map that Grails uses for domain object validation errors.

Normally, a grails hibernate session executes something quickly and returns. During importing, we do a ton of processing, all with the same hibernate session. All of these objects that would normally be garbage collected when the session closed are piling up.

The easiest way to deal with this is to create a simple method to clear out these collections periodically.

We can modify our BookService to clean up GORM after every 100 books we insert:

class BookService {
 
    def sessionFactory
    def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP
 
    def importBooksInLibrary(library) {
        library.eachWithIndex { Map bookValueMap, index ->
            updateOrInsertBook(bookValueMap)
            if (index % 100 == 0) cleanUpGorm()
        }
    }    
 
    def cleanUpGorm() {
        def session = sessionFactory.currentSession
        session.flush()
        session.clear()
        propertyInstanceMap.get().clear()
    }
 
    def updateOrInsertBook(Map bookValueMap) {
        // ... same as above
    }
}

This change makes everything much better. All 100,000 books get inserted in 3 minutes, the same time that it took the naive example to insert the last 1,000 books.

Also, because MySQL is now properly using indexes, the load on it is very light. If we’re looking for even more speed improvement, this would allow us to split our work up into batches and to make multiple BookService calls (potentially on different web servers). Because we’ve switched to using InnoDB, we’re now working with row-level locking, which isn’t hostile to this approach unlike MyISAM and it’s full table locking on Insert/Update.

Conclusion

This post is just some of what can be done to improve batch performance with Grails and MySQL. Using these tips, I was able to speed up my own imports to a level that met the speed requirments for our problem space, but kept our code much more maintainable. Our domain object remain the canonical code representation of our database model.

There are a number of other things that could be looked at to further speed up batch performance and that could be important to your particular application/infrastructure.

• connectivity between app server and DB
• giving enough memory to your app and DB servers
• other load into database and app servers
• anemic hardware
• poor database design

Be sure to weigh the cost of making changes in each of these areas, and run benchmark tests before committing to them. You’ll often find that what you think is the bottleneck, isn’t. Speed is a feature, but you need to balance it with the costs to achieve it.

For those that aren’t familiar with the plugin, it make creating testing data easy.

It automatically adapts to changes in your domain classes and makes your integration tests easier to write and much less fragile.

Resources

• Documentation on BitBucket
• Build-Test-Data Presentation

Overall, the plugin has been very stable for the past few releases and I’m thinking about just bumping the version up to 1.0 fairly soon. The plugin seems pretty feature complete when balanced with it’s ease of use. Most of the other features I’ve thought of would explode the complexity level of the plugin and defeat it’s purpose. Might be best to leave well enough alone for now :).

I had a hell of a time with the current grails release-plugin cycle (6 attempts before it finally went through). The grails release-plugin command uses an old version of svnkit that’s only compatible with svn 1.5. I’ve long since upgraded to 1.6 on my mac, though luckily I did it through macports and still had the old version available to activate (sudo port deactivate subversion && sudo port activate subversion @1.5.6_0).

I did some searching around to find out how to install an old version of subversion, but didn’t come up with anything after a quick google so I might be screwed after my wipe/reinstall for snow leopard next week till Grails patches things with svnkit 1.3.

I did open up a JIRA ticket for grails to upgrade to svnkit 1.3, and I was surprised that there wasn’t already a ticket open on it (and that trunk still had svnkit 1.2 in it).

Groovy Config DSL Overview

In Java, configuration is normally done with properties files. They’re kind of a pain because they’re inflexible, don’t allow executable code, and don’t easily provide a heirarchy.

Groovy greatly improves on Java by building in support for a simple configuration DSL using the ConfigSlurper.

To use it, just instantiate a new ConfigSlurper and give it the name of a groovy script file that you’d like it to parse.

// SampleConfig.groovy, a simple groovy script
person {
    firstName = "Ted"
    address {
        city = "Minneapolis"
    }
}
 
// TestConfiguration.groovy
def configObject = new ConfigSlurper().parse(SampleConfig)

This creates a ConfigObject, which is a subclass of a Map and can be similarly accessed using dot notation

println configObject
// prints: [person:[firstName:Ted, address:[city:Minneapolis]]]
 
assert "Minneapolis" == configObject.person.address.city

ConfigSlurper is built into groovy and is heavily used for Grails configuration. Grails is a “convention over configuration” framework, but anytime you want to modify the conventions, you’re likely going to be touching a config file. Grails plugin authors also often use config files to drive plugin behavior.

In larger projects, these config files can start to get quite large and unwieldy.

How ConfigSlurper Does It’s Thing

DSLs in groovy, like the ConfigSlurper DSL, work by intercepting calls to methods and property setters. If you look at the internals of the ConfigSlurper.parse(Script script, URL location) method (the big one at the bottom), you’ll see that it heavily messes with the metaClass of the configuration script by creating custom versions of invokeMethod and getProperty.

Intercepting Methods

When you have a config script that says:

person {
    firstName = "ted"
}

What you’re saying is call the “person” method and pass the { firstName = "ted" } closure to it.

Normally, this would blow up because you haven’t defined a “person” method. With the modified metaClass, invokeMethod instead adds “person” as a key in the config map and then runs the closure associated with person. Any subsequent method calls or properties in that closure are assigned as the value in the map to the “person” key.

Intercepting Property Assignments

Property assignment (like firstName = "ted") works a little differently. Every script has something associated with it called a binding. The binding holds script-wide variables that aren’t explicitly declared. When parsed by ConfigSlurper, calls that would normally set variables in the script’s binding are monitored and inserted into the map at the appropriate place in the stack.

So with the modified metaClass and binding the config above turns into this when slurped:

[person:[firstName:ted]]

If we had explicitly declared the name property, it wouldn’t get in the binding, and wouldn’t be part of the resulting config map.

person {
    def firstName = "ted"
}
 
// result: [person:[:]]

Modularizing Config Scripts

I got to thinking about how nice gsp files are to manage with the ability to include other gsp fragments. It would be great if we could do the same kind of thing with config files. That would let us better manage large config files. It would also let us define external config files that we pull in at run time, or config templates that we want to override just a piece or two out of. All from inside the config file itself.

Something like:

// SecurityConfig.groovy
security {
    includeScript( SecurityDefaults )
    active = true
}
 
// SecurityDefaults.groovy
username = "mysql"
password = "sekr1t"	    
url {
    port = 2112    
}

Loading SecurityConfig would have everything in SecurityDefaults and also set active = true:

def configObject = new ConfigSlurper().parse( SecurityConfig )
 
println configObject
// [security:[username:mysql, password:sekr1t, url:[port:2112], active:true]]
 
assert configObject.security.active == true
assert configObject.security.url.port == 2112

A Solution

Getting the same functionality in config files ended up being a bit more difficult a problem to solve than I first expected. A big part of that was due to some misconceptions I had about how the ConfigSlurper was working as a DSL, as well as what exactly a groovy Script was and how it differed from a regular Groovy class.

It turns out that when a groovy script is compiled, it’s contents are placed inside a concrete implementation of the abstract Script.run() method. So this:

// MyGroovyScript.groovy
person {
    firstName = "Ted"
}

Is functionally equivalent to this:

class MyGroovyScript extends Script {
    def run() {
	person {
     	    firstName = "Ted"
        }
    }	    
}

So one way to solve the problem is to create an abstract subclass of script and implement the “includeScript” method. All “includeScript” needs to do is override the metaClass and binding of the child script so that it uses the same metaClass and binding as the parent script.

public abstract class ComposedConfigScript extends Script {
    def includeScript(scriptClass) {
        def scriptInstance = scriptClass.newInstance()
        scriptInstance.metaClass = this.metaClass
        scriptInstance.binding = new ConfigBinding(this.getBinding().callable)
        scriptInstance.&run.call()
    }
}

ConfigSlurper modifies the metaClass and binding before loading the parent class to keep track of things as the script executes, it doesn’t care who’s actually calling the methods.

This lets us do exactly what we want, the only limitation to this approach is that the parent script needs to explicitly state that it’s a class that extends ComposedConfigScript and implements the run() method:

// SecurityConfig.groovy
class SecurityConfig extends ComposedConfigScript {	        
    def run() { // normal contents of a config file go in here
 
        security {
            includeScript( SecurityDefaults )
            active = true
        }
 
    }
}
 
// SecurityDefaults.groovy
username = "mysql"
password = "sekr1t"	    
url {
    host = "example.com"
    port = 2112    
}
 
// OtherClass.groovy
def configObject = new ConfigSlurper().parse( SecurityConfig )
 
println configObject
// prints: [security:[username:mysql, password:sekr1t, url:[host:example.com, port:2112], active:true]]

Using this technique lets us modularize our config files. This helps us both better organize things as well as decouple pieces of information that don’t necessarily belong together (like log4j info and website url information).

I had tried a number of other techniques that would let me use includeScript without declaring an explict class (like using a DelegatingMetaClass for the Script class and static import) but wasn’t able to get them working to my satisfaction. Those solutions are great for most cases, but break down with all the meta-magic that’s already happening with the ConfigSlurper.

Another option that could be considered, would be to patch the ConfigSlurper implementation to directly allow includeScript functionality. I’d be happy to submit a patch that does this if people are interested in this functionality.

Lots of good questions from the people in attendance. Thanks to everyone for showing up.

Here’s a version of it on slideshow:

I also released a new bugfix version of build-test-data earlier today (0.2.2). Mostly fixed bugs, but there’s also a new config setting that lets you disable the plugin in specified environments (suggested by Scott Vlaminck @ refactr.

So far, the proposals in the comments look fairly clunky to me and include defining whole sets of new methods like “assertEqualsButContinue(“foo”, “foo”)”, chaning assertions together jQuery-style, or using lists to hold all of our assertions.

Wouldn’t it be a lot nicer if we could continue to use the same methods that we’re already using?

In groovy, enabling soft assertions is easy with a little closure magic.

Closures have a property called the “resolveStrategy” that determines where the code inside the closure should look to resolve method names. The default resolve strategy is OWNER_FIRST.

Closure closure = { println "foo" }
assert closure.getResolveStrategy() == Closure.OWNER_FIRST

OWNER_FIRST means that if something isn’t defined inside the closure (local scope), it should check the owner of the closure to see if it can resolve it. The owner of the closure is the declaring class or another closure that contains our closure.

Closures also have a property called the “delegate”. By default, the delegate is equal to the owner, but we have the ability to change the delegate to something else. We can also change the resolveStrategy for the closure so when something isn’t defined, the delegate is the first place that is checked.

class Original {
   def closure = { return value() }   
 
   def value = { return "value from owner" }
}
 
class SomethingElse {
    def value = { return "value from delegate" }
}
 
def obj = new Original()
 
assert "value from owner" == obj.closure()
 
// the owner is the enclosing class, and its the same as the delegate by default
assert obj.closure.owner.class == Original
assert obj.closure.owner == obj.closure.delegate
 
// but we can change the closure's delegate to something else if we want
obj.closure.delegate = new SomethingElse()
 
// by default, the closure looks at the owner before the delegate, value is unchanged
assert "value from owner" == obj.closure() 
assert obj.closure.getResolveStrategy() == Closure.OWNER_FIRST
 
// changing the closure's resolve strategy switches who gets to handle value()
obj.closure.resolveStrategy = Closure.DELEGATE_FIRST
assert "value from delegate" == obj.closure()

Groovy also has a special metaClass method called invokeMethod that allows you to manipulate any method calls that are invoked on that metaClass.

Using these two things in conjunction, we can write a small class that lets us collect JUnit assertion errors thrown by any method starting with “assert”.

class SoftAsserts {
    def oldDelegate
    def failedAssertions = []
 
    static softAsserts(closure) {
        new SoftAsserts().bundleAsserts(closure)
    }
 
    private bundleAsserts = { closure ->
        closure.resolveStrategy = Closure.DELEGATE_ONLY
        oldDelegate = closure.delegate
        closure.delegate = this
        closure()   
        if (failedAssertions) {
            throw new Exception("${failedAssertions.size()} failed assertions found:\n${failedAssertions.message.join('\n')}")
        }
    }
 
    def invokeMethod(String name, args) {
        if (name.startsWith("assert")) {
            try {
                return oldDelegate.invokeMethod(name, args)
            } catch (junit.framework.AssertionFailedError e) {
                failedAssertions << e
            }
        } else {
            return oldDelegate.invokeMethod(name, args)         
        }
    }
}

This test class does a static import of the SoftAsserts.softAsserts closure so it can use it in a natural way.

import grails.test.*
import static SoftAsserts.softAsserts
 
class BookTests extends GroovyTestCase {
    void testBookTrueAssertsPasses() {
        def book = [title: "Infinite Jest"]
 
        // all 4 assertions pass and we are able to refer to local variables 
        // like book and local methods like trueMethod() 
        softAsserts { 
            assertTrue true
            assertTrue trueMethod()
            assertNotNull book
            assertEquals "Infinite Jest", book.title
        }
    }
 
    void testBookFalseAssertsCollectsFailures() {
        def book = [title: "Infinite Jest"]
 
        // all 4 assertions throw errors, but all are collected before 
        // an exception is actually thrown
        softAsserts { 
            assertTrue "False is not true", false
            assertTrue "falseMethod() is not true", falseMethod()
            assertNull "Book is not null", book
            assertEquals "House of Leaves", book.title
        }   
    }
 
    def trueMethod() { true }   
 
    def falseMethod() { false }
}

The first test shows how a passing test can successfully refer to other variables and methods defined inside the test class. The second test fails, but it doesn’t fail till all 4 of the assertions are tried and collected. It then throws an error with the assertion failure count and the individual error messages:

Testcase: testBookFalseAssertsCollectsFailures took 0.371 sec
	Caused an ERROR
4 failed assertions found:
False is not true
falseMethod() is not true
Book is not null
expected:<[House of Leaves]> but was:<[Infinite Jest]>

It might be useful to also print out the individual stack traces for each assertion error, but this is left as an exercise for the reader.

In most situations, I like the fail fast behavior and limiting tests to exercising one feature at a time, but there have been a few test cases where something like this would’ve been nice.