Just noticed that I never put up a link to the presentation I gave at the ClojureMN group a few months ago on what I thought was interesting at ClojureWest 2012 in San Jose earlier this year. Overall a solid conference with a good mix of hardcore topics mixed with pragmatic and practical Clojure usage.

Alex Miller always puts on a great conference (he also runs the StrangeLoop conferences). StrangeLoop 2011 was the best conference I’ve been to so far and I’m looking forward to StrangeLoop 2012 next month.

I gave a presentation earlier today on Git at the Groovy and Grails GR8Conf US 2012 conference.

The GR8Conf was named for the 8 groovy-based technologies starting with the letter “G” that were popular when the conference first started 2 years ago (I think they were Groovy, Grails, Gradle, Griffon, Gant, GPars, Gaelyk, and…? Maybe GContracts or Geb?). I don’t think that Git was one of the 8 technologies the conference was named for, but it probably should be. All of the ones listed have repositories out on GitHub and you need to know Git to be able to contribute and checkout the source.

The presentation is titled “Git Core Concepts…or: how I learned to stop worrying and love the reflog” and it can be found out on GitHub.

There’s also a repository with all of the presentations collected for the entire conference that Shaun Jurgemeyer (one of the main conference organizers) is putting together.

Thanks to Shaun and everyone else for putting on a fun conference this year. It was fun seeing a lot of familiar people and putting some faces to those I’ve only interacted with virtually in the past.

The Grails Redis plugin lets you use Redis as a store for all kinds of data. I find it especially useful as a compliment to a primary datastore. I use Redis for it’s caching and set operation strengths and let relational databases, and other NoSQL databases leverage their strengths.

One limitation of Redis is that there isn’t an embedded version of it that’s easy to spin up and shut down for tests (like H2 is for databases). This can make testing more difficult as your unit tests might stomp on the data that you want to keep in Redis when you’re doing a grails run-app. Your CI server might also have multiple sets of tests running at the same time that could hit race conditions if they’re all trying to make assertions against Redis at the same time.

The solution that I’ve come up with is to leverage Redis Database Numbers. Unlike a SQL database, Redis doesn’t have schemas to partition data. Instead it buckets data into various database numbers. By default, all operations are against database “0″. The default Redis configuration comes with 16 database buckets (numbered 0..15), but most people only use the first one (database 0). We can use those other buckets to hold our test data.

The factory class below creates a new Redis service that’s configured to point to a database number falling within testDatabaseRange (here 8..15). When you ask it for a new instance, it increments a key kept in database 0 and gives you back the modulus value that falls within your specified range. This lets multiple test servers operate against the same Redis instance and makes it so that they don’t use the same database at the same time.

package grails.plugin.redis.test

import grails.plugin.redis.RedisService
import redis.clients.jedis.JedisPool
import redis.clients.jedis.Protocol
import redis.clients.jedis.JedisPoolConfig
import redis.clients.jedis.Jedis

class TestRedisServiceFactory {

    public static Range testDatabaseRange = 8..15

    public static String DEFAULT_TEST_HOST = "localhost"
    public static int DEFAULT_TEST_PORT = Protocol.DEFAULT_PORT

    public static RedisService getInstance(Map configMap = [:]) {
        Map defaultedConfigMap = [
                host: DEFAULT_TEST_HOST,
                port: DEFAULT_TEST_PORT,
                timeout: Protocol.DEFAULT_TIMEOUT,
                password: null,
                poolConfig: [:]
        ] + configMap

        if (!configMap.database) defaultedConfigMap.database = nextTestDatabase(defaultedConfigMap)

        RedisService redisService = new RedisService()
        JedisPoolConfig poolConfig = new JedisPoolConfig()

        defaultedConfigMap.poolConfig.each { configKey, value ->
            poolConfig."$configKey" = value
        }

        JedisPool jedisPool = new JedisPool(
                poolConfig,
                defaultedConfigMap.host as String,
                defaultedConfigMap.port as Integer,
                defaultedConfigMap.timeout as Integer,
                defaultedConfigMap.password as String,
                defaultedConfigMap.database as Integer
        )

        redisService.redisPool = jedisPool
        return redisService
    }

    public static int nextTestDatabase(Map configMap) {
        Jedis jedis = new Jedis(configMap.host as String, configMap.port as int)
        assert jedis.ping() == "PONG", "Unable to ping redis server at ${configMap.host}:${configMap.port}, make sure you've got redis running"

        // cycle through redis databases so that we won't have collisions where redis tests for 2 CI jobs use the same database #
        Integer counter = jedis.incr("test:nextRedisDatabase")

        return (counter % (testDatabaseRange.max() + 1 - testDatabaseRange.min())) + testDatabaseRange.min()
    }

    private TestRedisServiceFactory() {}
}

Now you can create a unit test, and get full interaction with Redis, without worrying about deleting data you might want to keep:

@TestMixin(GrailsUnitTestMixin)
class MyServiceUnitTests {

    RedisService redisService = TestRedisServiceFactory.getInstance()
    MyService myService

    @Before public void setUp() {
        redisService.flushDB()  // ensure that we're clean on setUp so no one else pollutes us
        myService = new MyService(redisService: redisService)
    }

    @After public void tearDown() {
        redisService.flushDB() // ensure that we don't leave anything around to pollute anyone else
    }

    @Test void testRedisServiceUnitTest() {
        redisService.foo = "bar"
        assert redisService.foo == "bar"
    }

    @Test void testMyServiceWithRedis() {
        ... // setup, possibly using redisService to insert data into redis that we want as part of test setup

        myService.someMethod()  // some method that calls into myService's redisService

        ... // test assertions, possibly using redisService to assert that expected modifications have been made
    }
}

The only case where you’d have a collision is if you had more than testDatabaseRange concurrent tests running, or if one server ran through many quick tests, and another had a very slow test. The solution to that is to just increase the number of databases that Redis allocates at startup to something much larger (like 1024) and changing the testDatabaseRange to work with that new range.

If you’ve got a test that passes when run by itself but fails when run with the rest of the tests in your test suite, you’ve almost definitely got a test pollution problem. Test pollution happens when the results of one test live past the tearDown of a test and impact the results of subsequent tests.

This can happen in any kind of environment that has side-effects, but in Groovy and Grails programming, is most often caused by making changes to a metaClass or singleton instance that you forget to remove when the test is done.

One other place where test pollution often rears it’s head is when a set of tests run fine in development, but when pushed to an integration server it fails. Often, this is because the developer is working on OSX, but the build server is on Linux and those platforms run the tests in a different order.

These issues can be a real pain to track down and the most reliable approach is to figure out what tests run before the failing test (so tests that could be causing the failure) and run them in smaller and smaller batches till you figure out what the minimal set of tests are that, when run with your test, causes the failure to happen. Remove any test and the tests all pass. (more…)

Grails 2 has a lot of great new unit testing features that make many test scenarios easier.

The grails documentation does an OK job of describing some of the new features, but there really wasn’t anywhere that I could find that had a comprehensive list of changes you should make to your code when migrating from grails 1.3.X to 2.0.X.

This blog post is the list of changes that I wished I had when I started to migrate our code. (more…)