The grails documentation does an OK job of describing some of the new features, but there really wasn’t anywhere that I could find that had a comprehensive list of changes you should make to your code when migrating from grails 1.3.X to 2.0.X.

This blog post is the list of changes that I wished I had when I started to migrate our code.

I found these pages helpful in figuring things out:

• The Grails 2 Testing Documentation
• Peter Ledbrook’s “Upgrade to Grails 2″ blog post
• The Grails 2 “Upgrading From Previous Versions of Grails” section of the user guide.
• Rob Fletcher’s Upgrading to Grails 2: Part 1
• This google plus thread on people’s experiences upgrading to grails 2
• GRAILS-7617 – TestFor(ControllerClass) annotation doesn’t provide renderArgs

The company I’m consulting at made a large investment in unit testing in the grails 1.2/1.3 days with 500-600 unit tests that heavily leverage custom extended base classes, `Mixin`s, and the grails 1.3 way of doing unit testing. This investment helped us keep most of our testing quick in unit tests, but has made this the most painful grails upgrade cycle I’ve been through (more painful than the grails 1.0->1.1 upgrade, which was the previous worst).

There have been surprisingly few changes to the actual production code to get things going, the vast majority of changes have been in how unit testing works.

General Test Upgrade Tips

• If you use an IDE with a debugger, it’s very useful to have a separate copy of your code that works against grails 1.3.7 so that you can have 2 instances of IntelliJ and can step through line by line in each codebase to see where things diverge.
• Methods with @Test, @Before, and @After MUST be declared public void or else they won’t be picked up
• I had real problems with using the new grails annotations (like @Mock) while extending a custom super-class, I haven’t dug to the bottom of this yet, but the grails annotations seem to have trouble with class hierarchies, avoid them if possible
• To that end, if you use the new grails annotations (like @TestFor), you MUST remove the old extends GrailsUnitTestCase
• Integration tests should also never extend GrailsUnitTestCase, they should instead extend GroovyTestCase and just use assert
• I’ve hit a number of test pollution issues when refactoring my code, where one test hasn’t properly cleaned up after itself affecting other tests downstream. If your test passes in isolation, you’ll need to run the tests in the same order as grails test-app does and slowly take out tests till your tests passes. Then you know which test is causing the pollution. The grails 2 test runner doesn’t show you the test order, but you can get it by setting a breakpoint in the JUnit4GrailsTestType.doPrepare method (line 54 as of grails 2.0.3) and executing this in your IDE: testClasses.name.each { println it }. Then you can take that list, join it together into one line and prepend grails test-app on it to run them in order.

Another way to get the in-order list of test classes from the test reports after a failed test using a little grep and sed:

grep testsuite target/test-reports/TESTS-TestSuites.xml | grep -v testsuites | cut -d\  -f8-9 | sed -E 's/name="(.*)" package="(.*)"/\2.\1/' | grep .

Mocking Class Methods by mucking with the metaClass

Groovy is now more stringent on closure arguments matching the type signature of a method it’s meant to mock. If the method you’re mocking types it’s variables, you need to match that with your closure.

grails 1.3.x:

MyDomain.metaClass.'static'.executeUpdate = { theQuery, theParams -> return}

Grails 2.0.x:

MyDomain.metaClass.'static'.executeUpdate = { String theQuery, Collection theParams -> return}

Otherwise it won’t find your closure and will still call the regular method you’re trying to mock. Put a breakpoint/println in your mocked methods to ensure that they’re being called by the code under test.

Unit Testing Domain Classes in Grails 2

mockDomain

grails 1.3.x:

class MyTests { 
    ...
    mockDomain(Person)
    mockDomain(Vehicle)

Grails 2.0.x:

@Mock([Person, Vehicle])
class MyTests {
   ...

If you’ve given mockDomain instances to save, you’ll need to save them manually now.

grails 1.3.x:

mockDomain(Person, [new Person(name: "Bob")])

If you keep the old style of domain mocking through mockDomain you need to ensure that you aren’t calling it twice on the same class (say in setUp and then again in a test method, or in a superclass and a subclass). This causes really hard to find test pollution errors downstream with the grails 2.0.3 code. It appears that it’s messing with the metaClass again and re-mocking it, but then when the test is done not fully cleaning up after both metaClass modifications. I consider this a bug as mockDomain should be idempotent; if you’ve already mocked a class calling mockDomain again shouldn’t remock the class, but this is what we have now.

Grails 2.0.x:

@TestMixin(GrailsUnitTestMixin)
@Mock([Person])
class MyTests {
    ...
    @Test public void testPerson() {
        new Person(name: "Bob").save(failOnError: true)
    ...

One caveat is that validation is now enforced on these objects, you can either pass in validate:false on the object, or use the grails BuildTestData plugin to create your valid objects:

@TestMixin(GrailsUnitTestMixin)
@Build([Person])
class MyTests {
    ...
    @Test public void testPerson() {
        Person.build(name: "Bob")
        ...

The Build Test Data plugin has recently been enhanced to work with grails unit tests, and I recommend using it’s @Build annotation in place of @Mock in pretty much every case. It just makes things so much easier by building fully valid objects automatically for you.

Controller Unit Tests in Grails 2

mockConfig

grails 1.3.x:

mockConfig """
    grails.foo.bar='baz'
"""

Grails 2.0.x:

grailsApplication.config.grails.foo.bar='baz'

mockParams

grails 1.3.x:

mockParams.foo = "bar"

Grails 2.0.x:

params.foo = "bar"

So you can replace all mockParams with just params.

mockFlash

grails 1.3.x:

assertEquals "Expected Message", mockFlash.error

Grails 2.0.x:

assert "Expected Message" == flash.error

So you can replace all mockFlash with just flash.

renderArgs and redirectArgs

The very useful renderArgs and redirectArgs mock objects are gone in grails 2.0 and I find the new way of testing without them to be harder than it was.

renderArgs.view

grails 1.3.x:

assert 'edit' == renderArgs.view

Grails 2.0.x:

assert view.endsWith('/edit')

redirectArgs

grails 1.3.x:

assertEquals 'person', redirectArgs.controller
assertEquals 'show', redirectArgs.action
assertEquals personInstance.id, redirectArgs.id

Grails 2.0.x:

assert response.redirectUrl.endsWith("/person/show/${personInstance.id}")

(or just use contains or .split('/') to find parts of the url)

renderArgs.model

grails 1.3.x:

assert renderArgs.model.person.id == person.id

Grails 2.0.x:

assert model.person.id == person.id

This seems to work unless you’re rendering a template in your controller rather than a view. In that case, the model isn’t populated and cannot be accessed as far as I can tell. This is documented in GRAILS-8659 which is marked as “Not a Bug”, something I disagree with as the workaround is much more painful.

The workaround, per Graeme Rocher on that ticket is to mock the template rendering:

views['/test/_bar.gsp'] = 'Hello ${name}'
controller.renderTemplate()
assert response.contentAsString == "Hello John"

Where controller:

def renderTemplate() {
    render template:"bar", model:[name:"John"]
}

The naming convention of what you set in the map should be in the format “/person/_editAddress.gsp”, where you’re in the PersonController and the template name is editAddress but there were a few situations where I had trouble getting this right.

If you want to be sure what value to set the path to in the views map, set a breakpoint in RenderDynamicMethod.java around like 319 where it does this check:

String templateUri = webRequest.getAttributes().getTemplateURI(controller, templateName);

The value of that templateUri is what you want to set in your views map.

redirectArgs.params

grails 1.3.x:

assert "bar" == redirectArgs.params.foo

Grails 2.0.x:

// strip off everything up till the ? that marks the start of the query string
def params = WebUtils.fromQueryString(response.redirectUrl.find(/[^?]*$/))
assert "bar" == params.foo

gross. This really should be better and usability has regressed on this.

mockResponse

grails 1.3.x:

assertEquals 404, mockResponse.status
assertEquals "foo bar baz", mockResponse.contentAsString

Grails 2.0.x:

assert 404 == response.status
assertEquals "foo bar baz", response.contentAsString

So you should be able to find/replace mockResponse with just response

mockLogging

grails 1.3.x:

mockLogging(Foo)

grails 2.0.x:

In many cases, this can simply be deleted. If it’s either the class under test (the class you are doing a @TestFor on) or something you’ve already got in a @Mock or a @Build class level annotation, it already has logging.

For those classes that need logging, but are not already mocked, you can use the grails.test.MockUtils.mockLogging static method. Just add this static import to your class to get the same behavior:

import static grails.test.MockUtils.*
    ...
    mockLogging(MyService)

If it’s a collaborator that isn’t already mocked out, you can use the mockFor method as described in the mocking collaborators section of the grails user docs.

def foo = mockFor(Foo, true) // true here is "loose" mocking, otherwise you need to `demand` method calls for them to not fail.

Overall, I’m much happier with grails 2.0. It has a number of great features and speed improvements that make it a worthwhile upgrade. I complain above about a few places where I think testing is not as easy as it was, but most of these could be remedied with a few patches to grails-core.

brew install gradle

It’s very easy to use with a little experience, but I find having a good starting place to base your work from can help.

Here’s a quick function that I’ve got in my .zshrc to bootstrap a new groovy gradle project in the current directory (it should also work in in a .profile/.bash_profile/.bashrc).

function newgradle() {
    echo "Creating files for new gradle project"
 
    cat <<EOF>.gitignore
*.un~
*.iml
*.ipr
*.iws
build
.gradle
EOF
 
    cat <<EOF>build.gradle
apply plugin: 'groovy'
apply plugin: 'idea'
 
repositories {
    mavenCentral()
}
 
dependencies {
    groovy 'org.codehaus.groovy:groovy-all:1.8.6'
    groovy 'org.apache.ivy:ivy:2.2.0'
    testCompile 'junit:junit:4.10'
}
 
task createSourceDirs(description : 'Create empty source directories for all defined sourceSets') << {
    sourceSets*.allSource.srcDirs.flatten().each { File sourceDirectory ->
        if (!sourceDirectory.exists()) {
            println "Making \$sourceDirectory"
            sourceDirectory.mkdirs()
        }
    }
}
 
idea {
    project {
        jdkName = '1.6'
    }
}
EOF
 
    gradle createSourceDirs
 
    git init
    ls -a1 && find src    # list all created assets
}

It creates a build.gradle file ready to work with java and groovy projects, including IDEA integration (just execute gradle idea).

This gives you all of the tasks necessary to compile, jar, test, and distribute your code. For more information, check out the gradle docs on the java, groovy, and idea tasks.

It also creates all the necessary source directories for you and initializes a new git repository (with starting .gitignore file) for you to save your work.

You can easily tweak the build.gradle or .gitignore files to fit your needs. If you don’t use git, you can either delete those lines, or subsitute the equivalent lines for the source control tool you use. These are just a good starting place for me.

Here’s the sample output from the script above:

% mkdir testapp
% cd testapp
% newgradle                                                                   
Creating files for new gradle project
:createSourceDirs
Making /Users/tnaleid/Documents/workspace/testapp/src/main/resources
Making /Users/tnaleid/Documents/workspace/testapp/src/main/java
Making /Users/tnaleid/Documents/workspace/testapp/src/main/groovy
Making /Users/tnaleid/Documents/workspace/testapp/src/test/resources
Making /Users/tnaleid/Documents/workspace/testapp/src/test/java
Making /Users/tnaleid/Documents/workspace/testapp/src/test/groovy
 
BUILD SUCCESSFUL
 
Total time: 2.344 secs
Initialized empty Git repository in /Users/tnaleid/Documents/workspace/testapp/.git/
.
..
.git
.gitignore
.gradle
build.gradle
src
src
src/main
src/main/groovy
src/main/java
src/main/resources
src/test
src/test/groovy
src/test/java
src/test/resources

Then you’ve got all this gradle functionality ready to use:

% gradle tasks                                                           
:tasks
 
------------------------------------------------------------
All tasks runnable from root project
------------------------------------------------------------
 
Build tasks
-----------
assemble - Assembles all Jar, War, Zip, and Tar archives.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
classes - Assembles the main classes.
clean - Deletes the build directory.
jar - Assembles a jar archive containing the main classes.
testClasses - Assembles the test classes.
 
Documentation tasks
-------------------
groovydoc - Generates Groovydoc API documentation for the main source code.
javadoc - Generates Javadoc API documentation for the main source code.
 
Help tasks
----------
dependencies - Displays the dependencies of root project 'test'.
help - Displays a help message
projects - Displays the sub-projects of root project 'test'.
properties - Displays the properties of root project 'test'.
tasks - Displays the tasks runnable from root project 'test' (some of the displayed tasks may belong to subprojects).
 
IDE tasks
---------
cleanIdea - Cleans IDEA project files (IML, IPR)
idea - Generates IDEA project files (IML, IPR, IWS)
 
Verification tasks
------------------
check - Runs all checks.
test - Runs the unit tests.
 
Other tasks
-----------
cleanIdeaWorkspace
createSourceDirs - Create empty source directories for all defined sourceSets
 
Rules
-----
Pattern: build<ConfigurationName>: Assembles the artifacts of a configuration.
Pattern: upload<ConfigurationName>: Assembles and uploads the artifacts belonging to a configuration.
Pattern: clean<TaskName>: Cleans the output files of a task.
 
To see all tasks and more detail, run with --all.
 
BUILD SUCCESSFUL
 
Total time: 4.871 secs

We decided to leverage git’s history rewriting capabilities to make a much smaller repository (and kept our previous repo as a backup just in case).

Here are a few questions/answers that I figured out how to answer with git and some shell commands:

What object SHA is associated with each file in the Repo?

Git has a unique SHA that it associates with each object (such as files which it calls blobs) throughout it’s history.

This helps us find that object and decide whether it’s worth deleting later on:

git rev-list --objects --all | sort -k 2 > allfileshas.txt

Take a look at the resulting allfileshas.txt file for the full list.

What Unique Files Exist Throughout The History of My Git Repo?

If you want to see the unique files throughout the history of your git repo (such as to grep for .jar files that you might have committed a while ago):

    git rev-list --objects --all | sort -k 2 | cut -f 2 -d\  | uniq

How Big Are The Files In My Repo?

We can find the big files in our repo by doing a git gc which makes git compact the archive and stores an index file that we can analyse.

Get the last object SHA for all committed files and sort them in biggest to smallest order:

git gc && git verify-pack -v .git/objects/pack/pack-*.idx | egrep "^\w+ blob\W+[0-9]+ [0-9]+ [0-9]+$" | sort -k 3 -n -r > bigobjects.txt

Take that result and iterate through each line of it to find the SHA, file size in bytes, and real file name (you also need the allfileshas.txt output file from above):

for SHA in `cut -f 1 -d\  < bigobjects.txt`; do
echo $(grep $SHA bigobjects.txt) $(grep $SHA allfileshas.txt) | awk '{print $1,$3,$7}' >> bigtosmall.txt
done;

(there’s probably a more efficient way to do this, but this was fast enough for my purposes with ~50k files in our repo)

Then, just take a look at the bigtosmall.txt file to see your biggest file culprits.

Purging a file or directory from history

Use filter-branch to remove the file/directory (replace MY-BIG-DIRECTORY-OR-FILE with the path that you’d like to delete relative to the root of the git repo:

git filter-branch --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch MY-BIG-DIRECTORY-OR-FILE' --tag-name-filter cat -- --all

Then clone the repo and make sure to not leave any hard links with:

git clone --no-hardlinks file:///Users/yourUser/your/full/repo/path repo-clone-name

You can use this command from the parent directory that contains your git repository and it’s clone to see how much space each of them take, and how much you’ve shrunk the repo in size:

du -s *(/)     # add the -h flag to see the output in human readable size formats, just like ls -lah vs ls -la

With these commands, I was able to reduce the file size of our repo with a few thousand commits down below the size of the checked out repository (more than an order of magnitude smaller). I only removed old binary files, we still have full history for all code files.

There are no good-looking, “mac-like” merge tools, but if you swallow your pride there are a few different options for 3-way merges, including Araxis Merge ($$$!), DiffMerge, DeltaWalker, and FileMerge which comes free with XCode.

I’ve tried them all, and find them all confusing. They all tend use a 3-pane display to do the merging with your file in the left pane, the file you’re merging in the right pane, and the messy half-merged file in the middle.

That’s not enough information.

A 3-way merge actually has four important sources of information:

• LOCAL – your file with the changes you’ve made to it
• REMOTE – the file you’re merging in, possibly authored by someone else
• BASE – the common ancestor file that LOCAL and REMOTE came from
• MERGE_RESULT – the file resulting from the merge where you resolve conflicts

You often need to see all four of these pieces of information to make intelligent choices. Where you came from (LOCAL), where the other person’s changes came from (REMOTE), where you both started (BASE) and where you are now (MERGE_RESULT).

Most other 3-way merge tools either conflate or omit the BASE and that can make it harder to see what the right thing to do is.

Kdiff3 is my merge tool of choice. It’s not pretty; it’s cross-platform Qt based so it has a very old-school linux GUI feel to it. But like linux, it’s functional and can help you quickly be productive once you get over the learning curve.

If you’ve got 2 heads that you need to merge in your current repository:

We made this modification to file.txt:

diff --git a/file.txt b/file.txt
--- a/file.txt
+++ b/file.txt
@@ -1,1 +1,1 @@
-BASE: common_ancestor
+LOCAL: file in current working branch

and they made this change to the same file and line:

diff --git a/file.txt b/file.txt
--- a/file.txt
+++ b/file.txt
@@ -1,1 +1,1 @@
-BASE: common_ancestor
+REMOTE: other branch we're merging in

If you run your SCM’s merge command (here, in mercurial: hg merge -r 2), and have kdiff3 configured as your merge tool, you’ll get a pop-up window like this:

As you can see, it shows you all 4 pieces of information, BASE, LOCAL, and REMOTE on top, and the MERGE_RESULT file on the bottom. It currently has a Merge Conflict that you need to fix.

You can move from one unresolved conflict to the next using the triple up and triple-down colored arrows in the middle of the tool bar. When a conflict is highlighted, you can press any combination of the A, B, and C buttons in the toolbar. Pressing one of those buttons will resolve the conflict with the code from pane A, B, or C on top. So if the LOCAL file (your file) had the right changes in it, you’d press B.

It’s possible to press more than one button if code from multiple panes is valid. You can also directly edit the file in the MERGE_RESULT pane to make manual changes if the correct merge is not the exact text in A/B/C.

Another option, if you want to take all of the changes from one file and discard any changes from the others, is to go to the “Merge” menu and pick one of “Choose A Everywhere”, “Choose B Everywhere”, or “Choose C Everywhere”.

Once you’ve resolved your file, simply save it (cmd-S) and quit out of kdiff3. Your SCM should see the MERGE_RESULT no longer has any merge conflicts and will mark it as resolved, ready for you to commit it. If there are other files with merge conflicts, you can repeat the process with those files.

Installing kdiff3

Installing kdiff3 is as easy as downloading the latest version from sourceforge and copying it to your /Applications directory.

The app is a simple wrapper around a Qt-based application. It can be run from the command line at /Applications/kdiff3.app/Contents/MacOS/kdiff3. You can make a symlink of that into your path, but I assume in the instructions below only that you’ve got kdiff3 in your /Applications folder.

Mercurial Integration

Mercurial command line integration is pretty easy. Just open up your ~/.hgrc file (or create one if you don’t have it already), and add this to it:

[extdiff]
cmd.kdiff3 = /Applications/kdiff3.app/Contents/MacOS/kdiff3
 
[merge-tools]
kdiff3.args = $base $local $other -o $output

That configures kdiff3 as your merge tool of choice, so it should pop up automatically when you hit a merge conflict. You can also use it as a diff tool:

hg kdiff3 -r revision_to_compare_to

Git Command-Line Integration

To configure the git command line to use kdiff3 as a diff and merge tool, add this to your ~/.gitconfig:

[difftool "kdiff3"]
    path = /Applications/kdiff3.app/Contents/MacOS/kdiff3
    trustExitCode = false
[difftool]
    prompt = false
[diff]
    tool = kdiff3
[mergetool "kdiff3"]
    path = /Applications/kdiff3.app/Contents/MacOS/kdiff3
    trustExitCode = false
[mergetool]
    keepBackup = false
[merge]
    tool = kdiff3

Now you can use the external tool commands to look at diffs:

git difftool [revision_sha]

and fix unresolved merge conflicts:

git mergetool

Git Tower Integration

I’m normally a pretty hard-core command line user, but I still sometimes find the places I get myself in git confusing enough that I’m willing to use a GUI to get myself out. I think that git-tower is the best git GUI available right now.

It can show you diff files and lets you browse through history in it’s UI. It also lets you do a number of basic and mid-level commands (adding, committing, rebasing, squashing commits, cherry-picking, etc), but it doesn’t have a built-in diff or merge tool.

It does come with the ability to integrate with a few different merge tools out of the box, but not kdiff3.

After a little digging around, I found a generic post showing how configure Tower with a plist file and a shim shell script, but nothing specific to kdiff3.

I figured out how to get Tower and kdiff3 to play nice with each other, and have the instructions and files in a GitHub repository. To get it working, just clone the repository to a temporary directory, and run the install.sh script:

cd /tmp
git clone git@github.com:tednaleid/git-tower-kdiff3-shim.git 
cd git-tower-kdiff3-shim
./install.sh

It simply copies the plist file and the shim script into the appropriate directories (and Tower is very picky about those locations).

Then you need to go into Tower’s Preferences->Git Config control panel, and choose kdiff3 as the diff and merge tool:

For a quick rundown of how Tower works with kdiff3 as a merge tool, here’s an example where we’ve got one commit where we made changes to locally:

And another commit on origin/master where someone else made changes to the same piece of code:

We’ve pulled it down and now have 2 heads within our repository. To fix this, we need to merge and resolve the conflicts, hit the merge button and we’ll get an error message because there are conflicts that git can’t resolve automatically:

If you highlight a file with merge conflicts, and then hit the “Merge Tool” button, it will bring up kdiff3 and let us use it to resolve the issue:

Fix the merge conflicts in kdiff3 (ex: press “C” to change it to “Goodbye Cruel World”), save it, and quit out of kdiff3. Tower should see that the file has had it’s conflicts resolved and lets you commit the merge and carry on.

You can also use Kdiff3 as a regular diff tool if you don’t like looking at diff files. Just choose a file that you’ve modified to diff:

Press the “Diff Tool” button, and it’ll pop up in kdiff3:

Kdiff3 might be homely, but it’s easy to use once you understand how it works.

Install Leiningen

Leiningen isn’t required, but it makes managing the classpath for your REPL quite a bit easier and automates the inclusion of the files in your project so that they can be used in the REPL.

If you’ve already got clojure installed and are managing your classpath some other way, you’ll want to modify the nailgun server script below to remove the call to lein and ensure you’re getting clojure and other classes in the ng-server’s classpath.

Install Leiningen if you don’t have it already.

Download the lein script.

Put it somewhere on your PATH (such as /usr/local/bin or ~/bin).

Set it to be executable:

chmod +x lein

Then, tell it to self-install with this command:

lein self-install

Now you should have a working lein installation:

lein help

And you can create a new clojure project skeleton, with clojure 1.3.0 all ready to go with:

lein new myproj

It creates a skeleton project rectory and a project.clj file that you can use to hold lib dependencies.

Go into that project and you can use lein to go into a repl, get the project’s classpath, run tests, etc. Use lein help to see the full list of commands.

Install VimClojure in Vim

Install the VimClojure plugin in vim.

Hopefully, you’re already using Tim Pope’s vim script management script Pathogen.

If so, you should be able to just clone VimClojure into your bundle ectory:

cd .vim/bundle
git clone git://github.com/vim-scripts/VimClojure.git

If you’re not using Pathogen, install the plugin with your normal process.

Edit your .vimrc, make sure you have these settings if they’re not already there (they need to be after the method that calls pathogen to install bundles):

filetype off
call pathogen#runtime_append_all_bundles()
...
filetype plugin indent on
syntax on

and add these items to your .vimrc

let g:vimclojure#HighlightBuiltins = 1
let g:vimclojure#ParenRainbow = 1

To test that VimClojure is installed OK, edit a clojure file (like this sample one modified from (source +)):

(defn myplus
  ([] 0)
  ([x] (cast Number x))
  ([x y] (. clojure.lang.Numbers (add x y)))
  ([x y & more]
     (reduce myplus (myplus x y) more)))

If the parenthesis are differently colored in matched pairs, that means that “rainbow” parenthesis are enabled and the plugin is working. It should look something like this:

You’ll also want to update the helptags documentation for VimClojure:

:helptags ~/.vim/bundle/VimClojure/doc/

Now you can lookup help for VimClojure:

:help VimClojure

You’ve now got a functioning VimClojure installation that can already be pretty helpful. If you want to get an integrated REPL installed, there’s a bit more to do.

Setting up Nailgun to work with VimClojure

This part is the trickiest part (in the github docs, they say “Here be Dragons”). I think the biggest difficulty is that they don’t show you how to configure/launch the nailgun server, we’ll go through those steps.

Nailgun is a server that runs an instance of the JVM. On startup, you tell it what classpath this JVM should run with. Then, with a nailgun client, you can send it additional classes to execute within that JVM and it can show you the output.

It makes things much faster as the Nailgun server is already running hot when you send it things to execute. You don’t pay any JVM startup penalty.

Warning: If you’ve previously tried to install nailgun through homebrew, it will not work with clojure. The server jar that homebrew uses is half the size of the server jar that we’re going to be using for clojure. Uninstall it or put the things below earlier in the classpath.

Compile the VimClojure nailgun client

We need to compile the VimClojure nailgun client. You can either download the sources directly or else clone the bitbucket repo:

hg clone https://bitbucket.org/kotarak/vimclojure

Go into the client directory and make the ng client executable:

cd vimclojure/client  # or cd vimclojure-nailgun-client
make

This will create an executable called ng in the current directory. Copy that somewhere in your path (likely the same place you put lein above).

You should now be able to type ng and have it spit out help text.

% ng
Usage: ng class [--nailgun-options] [args]
          (to execute a class)
   or: ng alias [--nailgun-options] [args]
          (to execute an aliased class)
   or: alias [--nailgun-options] [args]
          (to execute an aliased class, where "alias"
           is both the alias for the class and a symbolic
           link to the ng client)
 
where options include:
   --nailgun-D<name>=<value>   set/override a client environment variable
   --nailgun-version           print product version and exit
   --nailgun-showversion       print product version and continue
   --nailgun-server            to specify the address of the nailgun server
                               (default is localhost)
   --nailgun-port              to specify the port of the nailgun server
                               (default is 2113)
   --nailgun-help              print this message and exit

Setting up Nailgun Server

Download latest server jar from clojars (currently 2.3.0).

Put that jar somewhere that you won’t delete it, and then add an export for that location in your .bashrc/.vimrc, ex:

export VIMCLOJURE_SERVER_JAR="$HOME/lib/vimclojure/server-2.3.0.jar"

Now save this script as ng-server and put it somewhere in your path and make it executable:

#! /bin/bash
 
if [ -z $VIMCLOJURE_SERVER_JAR ]; then
    echo "Error! Need to define location of VimClojure nailgun server jar with:"
    echo "export VIMCLOJURE_SERVER_JAR=<location>"
    exit 1
fi
 
if [ ! -f $VIMCLOJURE_SERVER_JAR ]; then
    echo "Error! Unable to find VimClojure nailgun server jar at '$VIMCLOJURE_SERVER_JAR'"
    exit 1
fi
 
LEIN_CLASSPATH=$(lein classpath)
 
if [ ! $LEIN_CLASSPATH ]; then
    echo "Warning! Unable to get classpath from lein, just using existing classpath, expecting clojure jars to be available"
fi
 
NG_CLASSPATH="$VIMCLOJURE_SERVER_JAR:$LEIN_CLASSPATH:$CLASSPATH"
echo java -server -cp "$NG_CLASSPATH" vimclojure.nailgun.NGServer
java -server -cp "$NG_CLASSPATH" vimclojure.nailgun.NGServer

Then add this to your .vimrc setup:

" this should only be necessary if you don't have the ng client in your PATH
let vimclojure#NailgunClient = "/path/to/your/ng"
let vimclojure#WantNailgun = 1

Now, from the root of a lein project, make sure you’re dependencies are updated:

lein deps   // puts things defined in project.clj into ./lib by default

Without additional configuration, you’ll likely just have the clojure-1.3.0.jar in there.

The script above leverages lein to create a classpath that the nailgun server can use. If you run lein classpath, you’ll see what it will use:

% lein classpath
/Users/tnaleid/Documents/workspace/clojure-test/myproj/test:/Users/tnaleid/Documents/workspace/clojure-test/myproj/test-resources:/Users/tnaleid/Documents/workspace/clojure-test/myproj/src:/Users/tnaleid/Documents/workspace/clojure-test/myproj/classes:/Users/tnaleid/Documents/workspace/clojure-test/myproj/resources:/Users/tnaleid/Documents/workspace/clojure-test/myproj/lib/clojure-1.3.0.jar

Now, you should be able to start up a nailgun server from the root of a lein project and have it automatically pull in all dependencies, ready to recieve client requests.

Open up the src/leinprojectname/core.clj file in vim. Then press

\sr

to get a REPL window that you can interact with. (if you’ve remapped localleader to something else, replace \ with your custom key)

There are quite a few other things you can do with the REPL, check out :help VimClojure and go down to Keybindings for more details.

Here are some good starter ones:

\el  - eval current line
\eb  - eval current visual block selected
\ef  - eval current file
 
\sr  - start interactive REPL
\sR  - start interactive REPL initialized to have same namespace as current buffer
 
\si  - prompt for input and lookup with (source)
\fd  - prompt for input and lookup with (find-doc)

The Pain is Over, Now Reap the Rewards

Once all this configuration is done, it should be as easy as running the server in a lein project:

ng-server

And then editing your files in vim accessing the embedded REPL whenever you need it.

Having to log in manually impedes development speed.

If you watch what your browser is doing when it’s interacting with a Spring Security application, you’ll see that (by default) it’s POSTing 2 parameters (j_username and j_password) to http://localhost:8080/YOURAPP/j_spring_security_check.

It’s easy to automate the login process with a little bit of vanilla javascript. Edit this javascript url to replace YOURAPP, YOURUSERNAME, and YOURPASSWORD, then make a bookmark out of it in your browser:

javascript:(function(){var%20path='http://localhost:8080/YOURAPP/j_spring_security_check';var%20params={'j_username':'YOURUSERNAME','j_password':'YOURPASSWORD'};var%20form=document.createElement("form");form.setAttribute("method","POST");form.setAttribute("action",path);for(var%20key%20in%20params){var%20hiddenField=document.createElement("input");hiddenField.setAttribute("type","hidden");hiddenField.setAttribute("name",key);hiddenField.setAttribute("value",params[key]);form.appendChild(hiddenField);}document.body.appendChild(form);form.submit();}());

Any time you want to log in, just click that bookmark. You’re now fully authenticated and in the app without having to interact with the login page.

Alternatively, if you’re using Google Chrome (or Firefox), you can create a “search engine” associated with a user-defined keyword. Type the keyword in the address bar to launch it.

You can even parameterize it to log in as a variety of users.

Say that you’ve got a number of different test users in your app: “admin”, “joeuser”, “sales”, “finance”, etc. All of the test users have the same password, but different usernames with different roles. If you make the username in the javascript url a “%s”, Chrome will replace that “%s” with your “search term”.

So if your app is “superapp” and all passwords are “password”, you can use this to create a Chrome search engine that lets you login with whatever test user you want

javascript:(function(){var%20path='http://localhost:8080/superapp/j_spring_security_check';var%20params={'j_username':'%s','j_password':'password'};var%20form=document.createElement("form");form.setAttribute("method","POST");form.setAttribute("action",path);for(var%20key%20in%20params){var%20hiddenField=document.createElement("input");hiddenField.setAttribute("type","hidden");hiddenField.setAttribute("name",key);hiddenField.setAttribute("value",params[key]);form.appendChild(hiddenField);}document.body.appendChild(form);form.submit();}());

To set it up, go into your preferences (cmd-,) and press the “Manage Search Engines” button.

Then under “Other Search Engines” click in the box to “Add a new search engine”

Name it with your app’s name (“superapp login”), set the keyword to an abbreviation of your app’s name (“sa”), and set the url to the edited javascript command to log in with your app’s url/username/password (potentially with the username as “%s” to parameterize it).

Once you save it, you can then go to your browser’s address bar (cmd-L) and type your abbreviation (“sa”) to get a new “search engine”. Then enter the username you want to log in as.

Hit enter and you’ll automatically be logged in to your app, without having to interact with your normal login page.

Automating this can help to keep you in the zone, especially if you’re using a security framework that allows deep linking.

If deep linking is enabled, the quickest way to get back to the page you’re iterating on after your session has expired (or you’ve bounced the app) is to reload the page. As it’s redirecting you to the login page, go to your address bar (cmd-L), type your keyword (ex: “sa”) and any associated username (ex: “admin”) and hit enter. You’ll be logged in before the login page displays and Spring Security will redirect you back to the page you originally requested.

I’ve recently needed to revisit some batch importing procedures and have acquired a few extra tools in my Grails utility belt since writing that post: Grails Redis and Grails Jesque.

Redis is a very fast key/value store, where the values are not just strings, but are data structures like lists, sets, and hash maps. I’m the main author of the grails redis plugin, and it’s my favorite pragmatic technology of the past few years. If you’re new to Redis, check out the presentation slides I gave at this year’s gr8conf.

Jesque is a Java implementation of Resque. A Redis-backed message queueing system for creating background jobs. The Jesque plugin is fully integrated with Grails and allows you to create worker jobs that are spring injected and have an active hibernate session. Resque was written in Ruby by the folks at GitHub.

This combination makes parallelizing work very easy, as most of the pain of trying to spin off threads in grails is handled for you by Jesque. Yes, there’s GPars, but the threads that it creates aren’t spring injected and don’t have hibernate sessions.

Using Jesque is as simple as:

1. create a Job class that implements a perform method.
2. tell Jesque to start up 1..n worker threads that monitor a queue and use your Job to process work
3. enqueue work on the queue so workers can pick it up

I’ve created a bitbucket repository with all of the source code from the original Batch Import post, as well as with the enhancements below.

The example problem is that there is a Library class that produces metadata for 100,000 books that we want to persist in the database as Book domain objects.

package com.naleid.example
 
class Book {
    String title
    String isbn
    Integer edition
 
    static constraints = {
    }
 
    static mapping = {
        isbn column:'isbn', index:'book_isbn_idx'
    }
}

The naive way of doing this takes Grails ~3 hours to do the inserts. The original batch performance post showed how to improve this time from 3 hours to 3 minutes with a few Grails and MySQL tweaks.

Using Redis + Jesque to parallelize the task, I’m able to cut that time in half again to a little over 90 seconds on my MacBook Air.

On real-world imports, where there is quite a bit more data and potentially other linked domain objects that can be memoized with the redis-plugin, I’ve seen a >100x speed improvement over the original serial import, even with the tuning tips from my original post.

Install redis and clone the test project from bitbucket to try it yourself. Just grails-run app, go to the running app on localhost and click on the link to the SerialBookController to see the original version, or the ParallelBookController to see the faster Redis+Jesque version. Each will display the length of time they took to do the insert after they’re done.

The ParallelBookController calls bookService.parallelImportBooksInLibrary(). That method spins up a number of worker threads, iterates through the books in the Library and enqueues each one on a Jesque queue. When it’s done iterating through the Library, it tells all the threads to end when they’re done processing all the work:

    def parallelImportBooksInLibrary(library) {
        Integer workerCount = 10
        String queueName = "import:book"
        withWorkers(queueName, BookConsumerJob, workerCount) {
            library.each { Map bookValueMap ->
                String bookValueMapJson = (bookValueMap as JSON).toString()
                jesqueService.enqueue(queueName, BookConsumerJob.simpleName, bookValueMapJson)
            }
        }
    }
 
    void withWorkers(String queueName, Class jobClass, Integer workerCount = 5, Closure closure) {
        def workers = []
        def fullQueueName = "resque:queue:$queueName"
        try {
            workers = (1..workerCount).collect { jesqueService.startWorker(queueName, jobClass.simpleName, jobClass) }
            closure()
            // wait for all the work we've generated to be pulled off the queue
            while (redisService.exists(fullQueueName)) sleep(500)
        } finally {
            // all work is off the queue, tell each worker to kill themselves when they're finished
            workers*.end(false)
        }
    }

The work queue that persist the Book domain objects in the database are very simple Jesque Job artefacts that are spring injected and have an active hibernate session. They can be of any class type. The only requirement is that they have a method named perform that is called and passed an item of work from the queue.

Here’s the example BookConsumerJob class that persists a Book to the database:

package com.naleid.example
 
import grails.converters.JSON
 
class BookConsumerJob {
    def bookService
 
    void perform(String bookJson) {
        bookService.updateOrInsertBook(JSON.parse(bookJson))
    }
}

You can see how simple the BookConsumerJob class is. It also calls out to the same bookService method that the serial batch import calls to import a Book.

One other neat thing about using Jesque is that it adheres to the Resque conventions for what gets stored in Redis. This means that you can gem install resque-web and then launch resque-web to get a nice monitoring platform for your Jobs and to see errors, or how much work is left in the queue.

One of my favorite uses of Dropbox is to sync almost all of the non-machine specific configuration files and directories in my home directory across all my OSX computers (currently my iMac, MacBook Air, and my work laptop).

Doing this lets me make a configuration change to one computer and have it almost instantly available on any other computer without any manual steps.

This is especially important for my zshell and Vim configurations as I’m always tweaking those, but it’s also helpful to have my Documents, Downloads and Pictures shared.

I have a folder in my Dropbox directory called home, I use a script called link.sh to automatically create symlinks in my home directory to the things I’ve got stored in Dropbox.

Dropbox/home currently has these files and directories in it:

.ackrc
.dbvis
.groovy
.gvimrc
.hg
.hgignore_global
.ssh
.subversion
.vim
.viminfo
.vimrc
.zshenv
Desktop-starling.local/   # unique Desktop for my MacBook Air
Desktop-kestrel.local/    # unique Desktop for my iMac
Desktop-thrush.local/     # unique Desktop for my work MacBook Pro
Documents/
Downloads/
Pictures/
bin/

My Dropbox/home directory also has a shell script in it called link.sh:

#! /usr/bin/env bash
cd $(dirname $0)
 
function linkFile() {
    LINK_TO_NAME=$2
    if [ -z $LINK_TO_NAME ]; then
        LINK_TO_NAME=$1
    fi
    if [ -a $HOME/$LINK_TO_NAME ]; then
        echo "**** Found existing $LINK_TO_NAME, skipping..."
    elif [ -h $HOME/$LINK_TO_NAME ]; then
        echo "Already symlinked $LINK_TO_NAME, skipping..."
    else
        echo "Linking $1 to $LINK_TO_NAME"
        ln -s $PWD/$1 $HOME/$LINK_TO_NAME 
    fi
}
 
 
for F in $(ls -a1 | grep -v link.sh | grep -v Desktop | egrep -v "^..?$" | egrep -v "^.*un~$" | grep -v .DS_Store); do
    linkFile $F
done
 
export HOSTNAME=$(hostname)
 
if [ -d "Desktop-$HOSTNAME" ]; then
    linkFile "Desktop-$HOSTNAME" "Desktop"
else 
    echo "Unable to find Desktop-$HOSTNAME to link to Desktop"
fi

What the script does is:

1. cd into the directory that the script is located in (it only symlinks files in the same directory)
2. list out all of the files and directories in the same directory as the script
3. filter out the things we don’t want to link (like ., .., the link.sh script itself, etc)
4. For all of the files/directories that pass the filter, call linkFile to create a symlink in the current user’s home directory as long as there isn’t already a file or a symlink there
5. Then look for a file called Desktop-$HOSTNAME where $HOSTNAME is the name of the current machine and create a ~/Dropbox symlink to it if it’s found.

It should be safe and non-destructive and only create symlinks when there isn’t anything else there with the same name.

I didn’t have my Pictures, Documents, and Downloads in my Dropbox for quite a while and was able to get away with the free 2GB plan. I recently upgraded to a paid Dropbox plan as I wanted those directories shared as well (though I exclude a couple of them from my work MacBook Pro).

For “special” directories like Desktop, Pictures, Documents, and Downloads, I needed to use sudo rm -r [dirname] to remove it before I could create the symlink (BACKUP THE DIRECTORY FIRST).

I’ve been using this for over a year, and haven’t noticed any apps that care that those directories are symlinks.

Also? I have used this shell script many times on my systems, and I think it’s safe, but PLEASE backup before using it, or deleting any directories. An adult crying is not a pretty sight :).

These aliases work for both the grails as well as the grails-debug commands (for attaching a remote debugger).

If there is an application.properties file in the current directory, we can find the current version of grails for the app.

If there isn’t an application.properties file in the current directory, the script just defaults to whatever version of grails you’ve already set up as your default through the standard $GRAILS_HOME environment variable. You can use the grails symlink switching aliases that I created previously to easily move this between versions.

alias grails="execute_grails_version grails"
alias grails-debug="execute_grails_version grails-debug"
 
function execute_grails_version() {
    GRAILS_CMD=$1
    shift
    if [ -f application.properties ]; then
        export GRAILS_VERSION=`grep app.grails.version application.properties | sed -E 's/.*=(.*)/\1/'`
        export GRAILS_HOME="/usr/local/grails-$GRAILS_VERSION"
        echo "application.properties found, using \$GRAILS_HOME of $GRAILS_HOME"
    else 
        echo "application.properties NOT found, leaving \$GRAILS_HOME as $GRAILS_HOME"
    fi
 
    if [ ! -d $GRAILS_HOME ]; then
        echo "ERROR: Unable to find \$GRAILS_HOME directory at $GRAILS_HOME"
        exit 1
    fi
 
    echo $GRAILS_HOME/bin/$GRAILS_CMD $*
    $GRAILS_HOME/bin/$GRAILS_CMD $*
}

UPDATE: There are a few situations where aliases aren’t available (or are a pain to get available) such as when code is being executed as part of another application rather than from the command line. To get around this, these scripts (created by a co-worker of mine, @sjurgemeyer) could be put in your PATH, ahead of your $GRAILS_HOME/bin and used instead of the aliases above:

grails:

grails-version grails $*

grails-debug:

grails-version grails-debug $*

grails-version:

GRAILS_CMD=$1
shift
if [ -f application.properties ]; then
    export GRAILS_VERSION=`grep app.grails.version application.properties | sed -E 's/.*=(.*)/\1/'`
    export GRAILS_HOME="/usr/local/grails-$GRAILS_VERSION"
    echo "application.properties found, using \$GRAILS_HOME of $GRAILS_HOME"
else 
    echo "application.properties NOT found, leaving \$GRAILS_HOME as $GRAILS_HOME"
fi
 
if [ ! -d $GRAILS_HOME ]; then
    echo "ERROR: Unable to find \$GRAILS_HOME directory at $GRAILS_HOME"
    exit 1
fi
 
echo $GRAILS_HOME/bin/$GRAILS_CMD $*
$GRAILS_HOME/bin/$GRAILS_CMD $*

The quick and dirty way to switch your logging level dynamically, if you know the name of the logger is just to do this in your console window:

import org.apache.log4j.*
Logger.getLogger("org.springframework").level = Level.DEBUG

Sometimes, a few helper methods can help you see what the current config is (especially if you’ve changed some things), as well as figure out what the right loggers are to tweak. This sample script can be used in a grails console to make it easy to view and change the logging level to whatever you want, just cut and paste it into your application’s console window (in dev it defaults to: http://localhost:8080/yourAppName/console):

import org.apache.log4j.Logger
import org.apache.log4j.Level
import static org.apache.log4j.Level.*
 
def getRootLogger() { Logger.rootLogger }
def getAllLoggers() { rootLogger.loggerRepository.currentLoggers.toList().sort { it.name } }
def getActiveLoggers() { allLoggers.findAll { it.level } }
def getLogger(String logName) { rootLogger.getLogger(logName) }
def setLevel(String logName, Level level) { rootLogger.getLogger(logName).level = level }
 
def printLogger(logger) { println "${logger.name} -> ${logger.level}" }
def printAllLoggers() { allLoggers.each { printLogger(it) } }
def printActiveLoggers() { activeLoggers.each { printLogger(it) } }

This makes it easy to see what logs are currently active (those with a log level set):

printActiveLoggers()

prints something like:

grails.app.filters.LoggingFilters -> DEBUG
grails.app.filters.SecurityFilters -> DEBUG
grails.app.service.grails.plugin.redis.RedisService -> WARN
grails.app.task -> DEBUG
org.apache.cxf -> DEBUG
...

You can also list all loggers, which also adds in those loggers who’s log level is currently `null`:

printAllLoggers()

prints:

grails.app -> DEBUG
grails.app.bootstrap.BootStrap -> null
grails.app.bootstrap.QuartzBootStrap -> null
grails.app.codec.org.codehaus.groovy.grails.plugins.codecs.Base64Codec -> null
grails.app.codec.org.codehaus.groovy.grails.plugins.codecs.HTMLCodec -> null
grails.app.codec.org.codehaus.groovy.grails.plugins.codecs.HexCodec -> null
...

You can also dynamically grab/create a logger and set it’s logging level to something more or less verbose than it’s current value:

def logger = getLogger("grails.app.service.grails.plugin.redis.RedisService")
printLogger(logger)   // initially WARN
logger.level = INFO  
printLogger(logger)   // prints INFO

prints:

grails.app.service.grails.plugin.redis.RedisService -> WARN
grails.app.service.grails.plugin.redis.RedisService -> INFO

It’d be easy to turn this into a simple gsp/controller that accepts changes and can list things out. There are also a number of other plugins out there that let you view/change logging levels (including another one of Burt’s plugins, app info), but if you don’t have those installed, this is a quick way to see what’s going on with your application.