Ted Naleid

The number of tests on my current project is getting close to the 2000 mark (1504 unit and 351 integration) and running the full suite can take 4+ minutes to execute.

When you’re trying to do TDD and want to run all tests before pushing your code out to the shared repository, this length of time can really bog you down when it happens a number of times per day.

There are a number of tests that we need to go back in and refactor to be faster, but I wanted to see how much of a boost I could get by running unit and integration tests in parallel.

With 2 threads (one for unit, one for integration) the best case would be taking 50% of the time that it takes to run serially. That would only happen if my unit and integration tests took the same amount of time to run. For my tests, I was able to get a 39% speed improvement (with the integration tests taking a bit longer than the unit tests).

Here’s the script:

(just save it as splitTests.groovy in your path and make it executable, then run it in your grails project dir)

#! /usr/bin/env groovy
 
Map exitStatuses = [:].asSynchronized()
 
def currentDir() { 'pwd'.execute().text.trim() }
 
def getGrailsPropertyArgs(group) {
	def testDir = "${currentDir()}/test-$group"
	[
	    "-Dgrails.work.dir=${testDir}",
	    "-Dgrails.project.class.dir=${testDir}/classes",
	    "-Dgrails.project.test.class.dir=${testDir}/test-classes",
	    "-Dgrails.project.test.reports.dir=${testDir}/test-reports"
	].join(" ")
}
 
synchronized out(group, message) {
    println("${group.padLeft(12, ' ')}: $message")
}
 
[
        integration: 'integration:',
        unit: 'unit:'
].collect { testGroup, args ->
	Thread.start {
		def command = "grails ${getGrailsPropertyArgs(testGroup)} test-app $args"
		out testGroup, command
		exitStatuses[testGroup] = command.execute().with { proc ->
			proc.in.eachLine { line -> out testGroup, line }
			proc.waitFor()
			proc.exitValue()
		}
		out testGroup, "exit value: ${exitStatuses[testGroup]}"
	}
}.each { it.join() }
 
def failingGroups = exitStatuses.findAll { it.value }
 
if (!failingGroups) {
	println "All tests were successful!"
} else {
	failingGroups.each { failingGroup, exitStatus ->
		out(failingGroup, "WARNING: '$failingGroup' exit status was $exitStatus")
	}
	System.exit(-1)
}

The script is pretty simple. It creates one thread for unit tests, and one for integration tests. It then creates a grails test-app command that includes a number of grails system properties necessary to define unique target and working directories so the threads don’t step all over each other.

As the threads run, they spit their output to the command line with a prefix on each line showing which thread it came from, ex:

...
 integration: Running test com.bloomhealthco.domain.ProductTests...PASSED
        unit: Running test com.bloomhealthco.domain.QuestionUnitTests...PASSED
 integration: Running test com.bloomhealthco.domain.TransactionalFalseTests...PASSED
        unit: Running test com.bloomhealthco.domain.RateAreaUnitTests...PASSED
        unit: Running test com.bloomhealthco.domain.RateDependentTypeUnitTests...PASSED
 integration: Running test com.bloomhealthco.filter.EmployerFiltersTests...PASSED
        unit: Running test com.bloomhealthco.domain.RateUnitTests...PASSED
...

When it’s done, it checks the exit status code of each individual run. If any failed, the splitTests script will enumerate the ones that failed and will also return a non-zero exit status code.

The first time you run this, it will probably be slow, as each thread is compiling it’s own set of class files into it’s own “target” directory. After the initial compilation, things should be a lot quicker for the 2nd run.

If you’ve only got a couple of tests, you likely won’t see any speedup from this as the fixed grails cost for boostrapping the environment in each thread will outweigh the parallelization of tests.

If you’ve got other types of tests that you’re running, it’s easy to just add other phases into the existing threads (or create new threads for them), just by adding to the map that we spin through.

I’ll probably dig into the grails internals some more to see if some of it can be refactored to make running tests in parallel easier. The latest versions of JUnit have some support for running tests in parallel, but it isn’t easily achieved with the current grails test/environment bootstrap.

My long term goal is to create a grails script that lets the user specify a number of worker threads to be used for testing. Those workers are then fed test classes to chew on and the load would be distributed better than it is currently.