Ted Naleid

Grails 2 has a lot of great new unit testing features that make many test scenarios easier.

The grails documentation does an OK job of describing some of the new features, but there really wasn’t anywhere that I could find that had a comprehensive list of changes you should make to your code when migrating from grails 1.3.X to 2.0.X.

This blog post is the list of changes that I wished I had when I started to migrate our code. Read the rest of this article »

When you’re doing dev on your website, how often do you log in with the same username and password? I bet it’s 20+ of times a day when you’re actively developing.

Having to log in manually impedes development speed.

If you watch what your browser is doing when it’s interacting with a Spring Security application, you’ll see that (by default) it’s POSTing 2 parameters (j_username and j_password) to http://localhost:8080/YOURAPP/j_spring_security_check.

It’s easy to automate the login process with a little bit of vanilla javascript. Edit this javascript url to replace YOURAPP, YOURUSERNAME, and YOURPASSWORD, then make a bookmark out of it in your browser:

javascript:(function(){var%20path='http://localhost:8080/YOURAPP/j_spring_security_check';var%20params={'j_username':'YOURUSERNAME','j_password':'YOURPASSWORD'};var%20form=document.createElement("form");form.setAttribute("method","POST");form.setAttribute("action",path);for(var%20key%20in%20params){var%20hiddenField=document.createElement("input");hiddenField.setAttribute("type","hidden");hiddenField.setAttribute("name",key);hiddenField.setAttribute("value",params[key]);form.appendChild(hiddenField);}document.body.appendChild(form);form.submit();}());

Any time you want to log in, just click that bookmark. You’re now fully authenticated and in the app without having to interact with the login page.

Alternatively, if you’re using Google Chrome (or Firefox), you can create a “search engine” associated with a user-defined keyword. Type the keyword in the address bar to launch it.

You can even parameterize it to log in as a variety of users.

Say that you’ve got a number of different test users in your app: “admin”, “joeuser”, “sales”, “finance”, etc. All of the test users have the same password, but different usernames with different roles. If you make the username in the javascript url a “%s”, Chrome will replace that “%s” with your “search term”.

So if your app is “superapp” and all passwords are “password”, you can use this to create a Chrome search engine that lets you login with whatever test user you want

javascript:(function(){var%20path='http://localhost:8080/superapp/j_spring_security_check';var%20params={'j_username':'%s','j_password':'password'};var%20form=document.createElement("form");form.setAttribute("method","POST");form.setAttribute("action",path);for(var%20key%20in%20params){var%20hiddenField=document.createElement("input");hiddenField.setAttribute("type","hidden");hiddenField.setAttribute("name",key);hiddenField.setAttribute("value",params[key]);form.appendChild(hiddenField);}document.body.appendChild(form);form.submit();}());

To set it up, go into your preferences (cmd-,) and press the “Manage Search Engines” button.

Then under “Other Search Engines” click in the box to “Add a new search engine”

Name it with your app’s name (“superapp login”), set the keyword to an abbreviation of your app’s name (“sa”), and set the url to the edited javascript command to log in with your app’s url/username/password (potentially with the username as “%s” to parameterize it).

Once you save it, you can then go to your browser’s address bar (cmd-L) and type your abbreviation (“sa”) to get a new “search engine”. Then enter the username you want to log in as.

Hit enter and you’ll automatically be logged in to your app, without having to interact with your normal login page.

Automating this can help to keep you in the zone, especially if you’re using a security framework that allows deep linking.

If deep linking is enabled, the quickest way to get back to the page you’re iterating on after your session has expired (or you’ve bounced the app) is to reload the page. As it’s redirecting you to the login page, go to your address bar (cmd-L), type your keyword (ex: “sa”) and any associated username (ex: “admin”) and hit enter. You’ll be logged in before the login page displays and Spring Security will redirect you back to the page you originally requested.

A couple of years ago, I put up a well-received blog post on tuning Batch Import Performance with Grails an MySQL.

I’ve recently needed to revisit some batch importing procedures and have acquired a few extra tools in my Grails utility belt since writing that post: Grails Redis and Grails Jesque.

Redis is a very fast key/value store, where the values are not just strings, but are data structures like lists, sets, and hash maps. I’m the main author of the grails redis plugin, and it’s my favorite pragmatic technology of the past few years. If you’re new to Redis, check out the presentation slides I gave at this year’s gr8conf.

Jesque is a Java implementation of Resque. A Redis-backed message queueing system for creating background jobs. The Jesque plugin is fully integrated with Grails and allows you to create worker jobs that are spring injected and have an active hibernate session. Resque was written in Ruby by the folks at GitHub.

This combination makes parallelizing work very easy, as most of the pain of trying to spin off threads in grails is handled for you by Jesque. Yes, there’s GPars, but the threads that it creates aren’t spring injected and don’t have hibernate sessions.

Using Jesque is as simple as:

1. create a Job class that implements a perform method.
2. tell Jesque to start up 1..n worker threads that monitor a queue and use your Job to process work
3. enqueue work on the queue so workers can pick it up

I’ve created a bitbucket repository with all of the source code from the original Batch Import post, as well as with the enhancements below.

The example problem is that there is a Library class that produces metadata for 100,000 books that we want to persist in the database as Book domain objects.

package com.naleid.example
 
class Book {
    String title
    String isbn
    Integer edition
 
    static constraints = {
    }
 
    static mapping = {
        isbn column:'isbn', index:'book_isbn_idx'
    }
}

The naive way of doing this takes Grails ~3 hours to do the inserts. The original batch performance post showed how to improve this time from 3 hours to 3 minutes with a few Grails and MySQL tweaks.

Using Redis + Jesque to parallelize the task, I’m able to cut that time in half again to a little over 90 seconds on my MacBook Air.

On real-world imports, where there is quite a bit more data and potentially other linked domain objects that can be memoized with the redis-plugin, I’ve seen a >100x speed improvement over the original serial import, even with the tuning tips from my original post.

Install redis and clone the test project from bitbucket to try it yourself. Just grails-run app, go to the running app on localhost and click on the link to the SerialBookController to see the original version, or the ParallelBookController to see the faster Redis+Jesque version. Each will display the length of time they took to do the insert after they’re done.

The ParallelBookController calls bookService.parallelImportBooksInLibrary(). That method spins up a number of worker threads, iterates through the books in the Library and enqueues each one on a Jesque queue. When it’s done iterating through the Library, it tells all the threads to end when they’re done processing all the work:

    def parallelImportBooksInLibrary(library) {
        Integer workerCount = 10
        String queueName = "import:book"
        withWorkers(queueName, BookConsumerJob, workerCount) {
            library.each { Map bookValueMap ->
                String bookValueMapJson = (bookValueMap as JSON).toString()
                jesqueService.enqueue(queueName, BookConsumerJob.simpleName, bookValueMapJson)
            }
        }
    }
 
    void withWorkers(String queueName, Class jobClass, Integer workerCount = 5, Closure closure) {
        def workers = []
        def fullQueueName = "resque:queue:$queueName"
        try {
            workers = (1..workerCount).collect { jesqueService.startWorker(queueName, jobClass.simpleName, jobClass) }
            closure()
            // wait for all the work we've generated to be pulled off the queue
            while (redisService.exists(fullQueueName)) sleep(500)
        } finally {
            // all work is off the queue, tell each worker to kill themselves when they're finished
            workers*.end(false)
        }
    }

The work queue that persist the Book domain objects in the database are very simple Jesque Job artefacts that are spring injected and have an active hibernate session. They can be of any class type. The only requirement is that they have a method named perform that is called and passed an item of work from the queue.

Here’s the example BookConsumerJob class that persists a Book to the database:

package com.naleid.example
 
import grails.converters.JSON
 
class BookConsumerJob {
    def bookService
 
    void perform(String bookJson) {
        bookService.updateOrInsertBook(JSON.parse(bookJson))
    }
}

You can see how simple the BookConsumerJob class is. It also calls out to the same bookService method that the serial batch import calls to import a Book.

One other neat thing about using Jesque is that it adheres to the Resque conventions for what gets stored in Redis. This means that you can gem install resque-web and then launch resque-web to get a nice monitoring platform for your Jobs and to see errors, or how much work is left in the queue.

I’m currently on a project that has a couple of different apps that are using different versions of grails that need to be run concurrently. Switching a symlink no longer fit the way I needed to work so I came up with a couple bash/zsh aliases that are smart about the version of grails for the current directory.

These aliases work for both the grails as well as the grails-debug commands (for attaching a remote debugger).

If there is an application.properties file in the current directory, we can find the current version of grails for the app.

If there isn’t an application.properties file in the current directory, the script just defaults to whatever version of grails you’ve already set up as your default through the standard $GRAILS_HOME environment variable. You can use the grails symlink switching aliases that I created previously to easily move this between versions.

alias grails="execute_grails_version grails"
alias grails-debug="execute_grails_version grails-debug"
 
function execute_grails_version() {
    GRAILS_CMD=$1
    shift
    if [ -f application.properties ]; then
        export GRAILS_VERSION=`grep app.grails.version application.properties | sed -E 's/.*=(.*)/\1/'`
        export GRAILS_HOME="/usr/local/grails-$GRAILS_VERSION"
        echo "application.properties found, using \$GRAILS_HOME of $GRAILS_HOME"
    else 
        echo "application.properties NOT found, leaving \$GRAILS_HOME as $GRAILS_HOME"
    fi
 
    if [ ! -d $GRAILS_HOME ]; then
        echo "ERROR: Unable to find \$GRAILS_HOME directory at $GRAILS_HOME"
        exit 1
    fi
 
    echo $GRAILS_HOME/bin/$GRAILS_CMD $*
    $GRAILS_HOME/bin/$GRAILS_CMD $*
}

UPDATE: There are a few situations where aliases aren’t available (or are a pain to get available) such as when code is being executed as part of another application rather than from the command line. To get around this, these scripts (created by a co-worker of mine, @sjurgemeyer) could be put in your PATH, ahead of your $GRAILS_HOME/bin and used instead of the aliases above:

grails:

grails-version grails $*

grails-debug:

grails-version grails-debug $*

grails-version:

GRAILS_CMD=$1
shift
if [ -f application.properties ]; then
    export GRAILS_VERSION=`grep app.grails.version application.properties | sed -E 's/.*=(.*)/\1/'`
    export GRAILS_HOME="/usr/local/grails-$GRAILS_VERSION"
    echo "application.properties found, using \$GRAILS_HOME of $GRAILS_HOME"
else 
    echo "application.properties NOT found, leaving \$GRAILS_HOME as $GRAILS_HOME"
fi
 
if [ ! -d $GRAILS_HOME ]; then
    echo "ERROR: Unable to find \$GRAILS_HOME directory at $GRAILS_HOME"
    exit 1
fi
 
echo $GRAILS_HOME/bin/$GRAILS_CMD $*
$GRAILS_HOME/bin/$GRAILS_CMD $*

If you’ve got Burt Beckwith’s great Grails Console Plugin installed, it’s easy to tweak the logging levels dynamically in your grails application.

The quick and dirty way to switch your logging level dynamically, if you know the name of the logger is just to do this in your console window:

import org.apache.log4j.*
Logger.getLogger("org.springframework").level = Level.DEBUG

Sometimes, a few helper methods can help you see what the current config is (especially if you’ve changed some things), as well as figure out what the right loggers are to tweak. This sample script can be used in a grails console to make it easy to view and change the logging level to whatever you want, just cut and paste it into your application’s console window (in dev it defaults to: http://localhost:8080/yourAppName/console):

import org.apache.log4j.Logger
import org.apache.log4j.Level
import static org.apache.log4j.Level.*
 
def getRootLogger() { Logger.rootLogger }
def getAllLoggers() { rootLogger.loggerRepository.currentLoggers.toList().sort { it.name } }
def getActiveLoggers() { allLoggers.findAll { it.level } }
def getLogger(String logName) { rootLogger.getLogger(logName) }
def setLevel(String logName, Level level) { rootLogger.getLogger(logName).level = level }
 
def printLogger(logger) { println "${logger.name} -> ${logger.level}" }
def printAllLoggers() { allLoggers.each { printLogger(it) } }
def printActiveLoggers() { activeLoggers.each { printLogger(it) } }

This makes it easy to see what logs are currently active (those with a log level set):

printActiveLoggers()

prints something like:

grails.app.filters.LoggingFilters -> DEBUG
grails.app.filters.SecurityFilters -> DEBUG
grails.app.service.grails.plugin.redis.RedisService -> WARN
grails.app.task -> DEBUG
org.apache.cxf -> DEBUG
...

You can also list all loggers, which also adds in those loggers who’s log level is currently `null`:

printAllLoggers()

prints:

grails.app -> DEBUG
grails.app.bootstrap.BootStrap -> null
grails.app.bootstrap.QuartzBootStrap -> null
grails.app.codec.org.codehaus.groovy.grails.plugins.codecs.Base64Codec -> null
grails.app.codec.org.codehaus.groovy.grails.plugins.codecs.HTMLCodec -> null
grails.app.codec.org.codehaus.groovy.grails.plugins.codecs.HexCodec -> null
...

You can also dynamically grab/create a logger and set it’s logging level to something more or less verbose than it’s current value:

def logger = getLogger("grails.app.service.grails.plugin.redis.RedisService")
printLogger(logger)   // initially WARN
logger.level = INFO  
printLogger(logger)   // prints INFO

prints:

grails.app.service.grails.plugin.redis.RedisService -> WARN
grails.app.service.grails.plugin.redis.RedisService -> INFO

It’d be easy to turn this into a simple gsp/controller that accepts changes and can list things out. There are also a number of other plugins out there that let you view/change logging levels (including another one of Burt’s plugins, app info), but if you don’t have those installed, this is a quick way to see what’s going on with your application.

I released the first version of the Grails Redis Plugin over the weekend. It’s a brand new plugin that takes the place of the previous redis plugin (which is being renamed to “redis-gorm” and has been refactored to use this plugin as a dependency). It’s version is 1.0.0.M7 just so it’s “higher” than the plugin it’s replacing, though I’d probably make it a 0.9 release if I were releasing it under a new name till I get a little more community feedback.

Quick description of what Redis is from the README:

The best definition of Redis that I’ve heard is that it is a “collection of data structures exposed over the network”.

Redis is an insanely fast key/value store, in some ways similar to memcached, but the values it stores aren’t just dumb blobs of data. Redis values are data structures like strings, lists, hash maps, sets, and sorted sets. Redis also can act as a lightweight pub/sub or message queueing system.

Redis is used in production today by a number of very popular websites including Craigslist, StackOverflow, GitHub, The Guardian, and Digg.

The Grails Redis plugin makes a Redis connection pool available (and injectable as a spring bean) to your Grails application.

To install the plugin, just execute this inside your application’s directory:

grails install-plugin redis

It allows you to transparently interact with your Redis instance by automatically handling retrieving a connection from the pool and ensuring that the connection is returned to the pool as it delegates to redis.

// overrides propertyMissing and methodMissing to delegate to redis
def redisService
 
redisService.foo = "bar"   
assert "bar" == redisService.foo   
 
redisService.sadd("months", "february")
assert true == redisService.sismember("months", "february")

One of the plugin’s greatest strengths is in the memoization methods and tag libraries that it adds. It’s a write-through cache (with optional TTL expiration). Before executing the closure/tag, it will check Redis to see if we’ve already calculated that value. If we have, we’ll just return the answer from Redis, otherwise, we’ll calculate it, and save it in Redis for future calls.

service method:

redisService.memoize("user:$userId:helloMessage") {
    // expensive to calculate method that returns a String
    "Hello ${security.currentLoggedInUser().firstName}"
}

taglib:

<redis:memoize key="mykey" expire="3600">
    <!-- 
        insert expensive to generate GSP content here 
 
        taglib body will be executed once, subsequent calls 
        will pull from redis till the key expires
    -->
    <div id='header'>
        ... expensive header stuff here that can be cached ...
    </div>
</redis:memoize>

Check out the full documentation on the github repository.

If you’re new to using Redis with Groovy, I created an introductory post and gave a presentation at gr8conf that are good starting places.

If you use OSX for development, you might also find these instructions for automatically launching Redis on startup with launchd useful.

A couple of weeks ago, I gave a talk on Redis, Groovy and Grails at the Groovy Users of Minnesota meeting. I’m giving that presentation again tomorrow at gr8conf 2011 and I wanted to post the slides so that people had access to them on the web.

An original version of the keynote file that I presented can be found on my bitbucket account. You can download the repo or grab the raw versions off bitbucket.

The grails console has a number of disadvantages that make it a little clunky to use:

• You can’t attach it to a running grails instance, you need to run grails console from scratch
• You can’t pass it parameters, such as the name of a script you’ve saved
• It’s a swing app, which is always a little wonky in it’s keybindings and it’s behavior
• It’s not <insert your chosen editor>

I got tired of these limitations so I decided to do something about it. I put together a groovy script that uses HTTPBuilder to POST groovy code to a running Grails app that has the grails console plugin installed.

If you’re running this in any non-development environment, you’ll want to ensure that it’s behind some form of authentication. The most popular grails security plugin is Burt Beckwith’s spring security core, so this script comes with built-in support for that.

This script makes it easy to call from any editor that allows you to execute scripts (pretty much every programmer’s editor on the market). At the bottom of the post, I show how to integrate it with Vim (my editor of choice), but it’d be just as easy to script it from TextMate, JEdit, IntelliJ, Eclipse, etc. Any editor that can take the file you’re currently editing and pass it to a shell script.

Here’s my Vim session with a grails script at the bottom and the output from that grails script in another window at the top.

Read the rest of this article »

When I’m integration testing Grails service classes, I often want to mock off a part of the class so that a complicated code branch isn’t followed that I’m not trying to test.

Grails will helpfully inject fully autowired Spring service beans into my test if I ask for them. Unfortunately, if I change the metaClass of the injected service, that change persists beyond where we want it to:

class MyService {
    def myMethod() { "unmodified" }
}
 
class MyServiceTests extends GroovyTestCase {
    def myService  // injected automatically by spring/grails
 
    void testOne() {
        myService.metaClass.myMethod = {-> "modified" }
        assertEquals "modified", myService.myMethod()
    } 
 
    void testTwo() {
        assertEquals  "unmodified", myService.myMethod() // WTF!  Returns "modified", pollution from first test
    }
}

Grails services are Spring “singleton” objects. They’re not true singletons though, singleton’s are just cached in the application context and returned whenever getBean is called.

Historically, if I wanted to mock part of my service manually, I’d need to “new” up my own instance of the service and manually inject any dependencies that the service might need to function. This is both painful and fragile, if the service adds or removes dependencies, chances are that the tests are going to break.

I realized that if I could ask Grails/Spring for a new instance of the “singleton” service that I could muck with it all I wanted in my test without worrying about polluting other tests with my changes. After some digging into the grails spring support, I came up with the following method that could be added to an integration test (or integration test base class):

// spring "singleton" objects really aren't they're just cached by their application context
def getNewSingletonInstanceOf(Class clazz) {
    String beanName = "prototype${clazz.name}"
    BeanBuilder beanBuilder = new BeanBuilder(ApplicationHolder.application.mainContext)
 
    beanBuilder.beans {
        "$beanName"(clazz) { bean ->
            bean.autowire = 'byName'
        }
    }
 
    beanBuilder.createApplicationContext().getBean(beanName)
}

This method uses the BeanBuilder to construct a temporary ApplicationContext with the Grails mainContext as a parent so that other dependencies can be resolved. This method won’t work if your service has changes to how it’s wired up and configured by spring, but the majority of Grails service classes are simply autowired byName.

Here’s a more detailed example of use. Given this service:

package com.example
 
class MyService {
    def injectedService
 
    def serviceMethod() {
        return otherMethod()
    }
 
    def otherMethod() {
        return "original value"
    }
}

I’m able to generate per-test autowired instances of my service and mock out otherMethod without polluting other tests (or the Spring injected version of the bean):

package com.example
 
import grails.spring.BeanBuilder
import org.codehaus.groovy.grails.commons.ApplicationHolder
 
class MyServiceTests extends GroovyTestCase {
    def myServiceInstance  // our new
    def myService // spring injected version
 
    protected void setUp() {
        super.setUp()
        myServiceInstance = getNewSingletonInstanceOf(MyService)
    }
 
    protected void tearDown() {
        super.tearDown()
    }
 
    // spring "singleton" objects really aren't they're just cached by their application context
    def getNewSingletonInstanceOf(Class clazz) {
        String beanName = "prototype${clazz.name}"
        BeanBuilder beanBuilder = new BeanBuilder(ApplicationHolder.application.mainContext)
 
        beanBuilder.beans {
            "$beanName"(clazz) { bean ->
                bean.autowire = 'byName'
            }
        }
 
        beanBuilder.createApplicationContext().getBean(beanName)
    }
 
    void testNewSingletonInstance() {
        assertNotNull myServiceInstance // created uniquely for this test in setUp
        assertNotNull myService         // spring injected into integration test
 
        assertNotSame myService, myServiceInstance
 
        // we've got unique instances of MyService, but both are injected with the same singleton dependencies
        assertSame myService.injectedService, myServiceInstance.injectedService
    }
 
    void testMessingWithMetaClassDoesNotAffectOriginalSingleton() {
        myServiceInstance.metaClass.otherMethod = {-> "new value" }
 
        assertEquals "new value", myServiceInstance.serviceMethod()
        assertEquals "original value", myService.serviceMethod()
 
        def anotherInstance = getNewSingletonInstanceOf(MyService)
 
        assertEquals "original value", anotherInstance.serviceMethod()
    }
}

Using this technique lets you leverage Spring’s autowiring in your tests, but also gives you the flexibility to override areas not under test to improve test readability and maintainability.

I’m working with an existing library (Jedis a Redis client library) that has a fairly complicated connection pool config file with a large variety of potential properties that could be worth setting depending on the environment that my Grails app is running in.

I wanted the ability to define the set of properties that I wanted to override in the config file without having to call them all out explicitly in the Spring resources.groovy file. If I missed one, or if the client library that I’m using added a new one that I don’t notice, I don’t want to have to release a new version of the code just to set it.

Grails allows you to load external config files simply by defining a reference to them in Config.groovy (this code is even commented out in the default Config.groovy file that gets generated automatically with a new grails app):

grails.config.locations = ["file:${userHome}/.grails/${appName}-config.groovy"]

After a little playing around with the BeanBuilder syntax, I was able to come up with a solution that lets me set whatever values I want in the Config file and have them set on the bean that I have Spring/Grails build.

If you have a config like this:

foo {
    foo = "bar"
    baz = 4
}

You can populate your resources.groovy with something like this to set whatever
values are set in your config file:

beans = {
    def fooMap = application.config?.foo
 
    fooBean(Foo) {
        fooMap?.each { key, value ->
            delegate.setProperty(key, value)
        }
    }
}

This will make a bean that has it’s `foo` set to “bar” and it’s `baz` set to 4.

Later, if I find that I need to set the `baz` property on the fooBean in production, I just add that in my config file and everything works without any code changes.