Since then, I’ve completely abandoned bash, in favor of zsh (which I consider to be the superior shell) and I’d been getting annoyed at a few issues in the last grails autocomplete script.

I finally got motivated to make some enhancements to it. Including support for grails 1.2 plugin scripts (1.2 moved the plugins into the ~/.grails directory), and support for test class name autocompletion (very useful for grails test-app).

To get it working (assuming you’re using zsh), you can either add the contents of my zshrc_grails_compinstall to your .zshrc file, or you can switch over to using my full zshrc setup, which has a number of nice features that I’ve collected over the years. I’ll also continue to update this as I think of new tricks.

After getting it installed, just type “grails” followed by a space and then hit tab. It will show you a list of all of the potential grails scripts for the application that you’re in. It’s aware of the version of the current app, as well as it’s application name based on the contents of application.properties, and also will include any scripts for the plugins you have installed in that app in ~/.grails/GRAILS_VERSION/projects/APP_NAME/plugins (in addition to the scripts in ./scripts, $GRAILS_HOME/scripts and ~/.grails/scripts).

After you choose your script (such as grails test-app hit space again and it will show you all of the test classes, with the full package for the class, under your test directory.

One of these changes is that each sheet can actually have multiple tables on it, and these can be arranged independently on the same page. This prevents the problem that happens in excel where you have multiple sets of data you want to see at the same time, but the cell/row sizing for one set of data affects the data in the other set that happens to be on the same row.

I’ve been leveraging this for a one-off personal project and I had a need to sum up all of the data in the 2nd column on all of the tables within a particular sheet.

This brought me to AppleScript, Apple’s scripting language that’s used to drive applications.

You can write scripts in the AppleScript editor (found in /Applications/Utilities). In there, you can look up the commands and objects that a particular program makes available by going to File->Open Dictionary and then finding the application that you want to script.

I’ve done a little work in AppleScript in the past, but always recoil from it when I’m done. It’s so damn wordy in what appears to be an attempt to make it more “friendly” when it really just makes it harder to remember the right syntax to get something done. It probably just has “too many notes”.

Regardless, here’s the script that I came up with to sum up all of the values in the 2nd column (Column B) for all of the tables on the currently active sheet and then insert the result onto the 2nd column of a table named “Totals” (which must previously exist).

tell application "Numbers"
	tell document 1
		set total to 0
		set totalTable to 0
		tell sheet 1
			repeat with j from 1 to (count of tables)
				try
					tell table j
						if (name is not equal to "Totals") then
							repeat with k from 1 to count of rows
								set total to total + (value of cell 2 of row k)
							end repeat
						else
							set totalTable to j
						end if
					end tell
				end try
			end repeat
			tell table totalTable
				display dialog "total = " & total & " total table = " & totalTable
				set value of cell 2 of row 1 to total
			end tell
		end tell
	end tell
end tell

I needed this ability as the number of tables on this sheet will change quite frequently depending on what I’m doing with it, and the values in the 2nd column of each of these tables will also get frequently modified and I need to easily know the sum across all of the tables.

To actually execute the script within Numbers.app, there are a couple of ways to do it. It doesn’t appear that Numbers has it’s own script directory (normally, it would be under ~/Library/Application Support/iWork/Numbers/Scripts), but you can add it to your own User Scripts directory at ~/Library/Scripts.

If you want to access it through your menus, you can enable the global script icon up there by going into the AppleScript Editor preferences. On the General tab, check the box that says “Show Script menu in menu bar”.

If you’re using LaunchBar or Quicksilver, you can also add this directory to their catalog as a place to search. Then it’s as easy as cmd-space and the name of the script to execute it (both apps also allow you to set up keyboard shortcuts too).

In all, it feels hacky, and I’m sure it could be improved if I actually knew what the hell I was doing in AppleScript, but as I said, it’s a one-off that I need for a fairly particular problem.

I should probably take a little more time to learn more about AppleScript (or look more closely at a bridge to a language that I find a bit more palatable). Being able to script the applications that I use can be very useful.

class ClosureInterrogator {
    private Map closureValueMap = [:]
 
    static Map extractValuesFromClosure(Closure closure) {
        def interrogator = new ClosureInterrogator(closure)
        return interrogator.closureValueMap
    }
 
    private ClosureInterrogator(Closure closure) {
        def oldResolveStrategy = closure.getResolveStrategy()
        def oldDelegate = closure.getDelegate()
        closure.delegate = this
        closure.resolveStrategy = Closure.DELEGATE_FIRST
 
        try {
            closure()
        } finally {        
            closure.setDelegate(oldDelegate)
            closure.setResolveStrategy(oldResolveStrategy)
        }
    }
 
    // property getter
    def propertyMissing(String name) {
        return closureValueMap[name]
    }
 
    // property setter
    def propertyMissing(String name, value) {
        closureValueMap[name] = value
    }
 
    def methodMissing(String name, args) {
        if (args.size() == 1) {
            closureValueMap[name] = args[0]
        } else {
            closureValueMap[name] = args
        }
    }
}

This technique is the basis for all groovy mocking techniques and libraries, it’s also the main technique used to create a DSL with groovy.

Here is a sample class that has a closure (“something”) that we want to extract the values from.

class SomeClass {
    static something = {
        key1 "value1"              // calls methodMissing("key1", ["value1"])
        key2("value2")             // calls methodMissing("key2", ["value2"])
        key3 = "value3"            // calls propertyMissing("key3", "value3")
        key4 "foo", "bar", "baz"   // calls methodMissing("key4", ["foo","bar","baz"])
    }
}

If we call “extractValuesFromClosure” on the “something” closure, we’ll get back a map that has all of the values we want in it.

def closureValueMap = ClosureInterrogator.extractValuesFromClosure(new SomeClass().something)
 
assert "value1" == closureValueMap."key1"  // calls propertyMissing("key1")
assert "value2" == closureValueMap."key2"  // calls propertyMissing("key2")
assert "value3" == closureValueMap."key3"  // calls propertyMissing("key3")
assert ["foo", "bar", "baz"] == closureValueMap."key4"  // calls propertyMissing("key4")

This class will work with any type of closure, such as this closure used by the grails mail plugin to send mail.

def mailClosure = {
   to "fred@g2one.com","ginger@g2one.com"
   from "john@g2one.com"
   cc "marge@g2one.com", "ed@g2one.com"
   bcc "joe@g2one.com"
   subject "Hello John"
   body 'this is some text'
}
 
def results = ClosureInterrogator.extractValuesFromClosure(mailClosure)
println results
// prints:
// [to:[fred@g2one.com, ginger@g2one.com], from:john@g2one.com, cc:[marge@g2one.com, ed@g2one.com], bcc:joe@g2one.com, subject:Hello John, body:this is some text]
 
assert results.to.every { it.endsWith("g2one.com") }

If we wanted to assert that within a test environment, we’re only sending e-mail to a specific domain, this would be one technique to assert it before we actually send the e-mail off.

ClosureInterrogator is generic enough that it could also be used on a number of Grails/GORM domain closures such as “constraints” and “mapping”.

I responded with a long reply that I thought would be worth cleaning up and posting here.

Mercurial, Bitbucket and Hudson

I’ve been using Mercurial as my personal revision control system for a while now.

I’ve more recently been using it at work with a repository hosted at BitBucket, a great service that offers free accounts (including one private repo) and very reasonable upgraded accounts. (I don’t have any relationship with bitbucket other than just as a happy and satisfied user.)

Being a good agile shop, we use hudson as our continuous integration server (much better than Cruise Control IMO). It polls the source control repository on a periodic basis, if it sees any code changes, it checks them out and runs all of the tests.

Quick Mercurial/Hudson Setup

Install Mercurial

There are lots of ways to install mercurial. On the Ubuntu box, the quickest way to get a recent version for me was to just use python’s easy_install:

sudo easy_install mercurial

Install Hudson

There are better guides out there for getting Hudson going depending on the platform you’re using, but here’s the quick instructions for how I got it going on an Ubuntu 9.10 image running on Amazon EC2.

Using Ubuntu’s apt-get package management, installing hudson is easy. You need to add the repository that has the hudson source so that apt-get knows where to get it. Add this to your /etc/apt/sources.list file:

deb http://hudson-ci.org/debian binary/

Then update apt-get and install hudson:

sudo apt-get update
sudo apt-get install hudson

That creates a hudson install with a new “hudson” user on your system.

Setting up Hudson to SSH

We want this user to be able to connect to BitBucket over SSH and check out the repository. To do this, we’ll need an ssh public and private key that hudson can use. Sudo to the hudson user and generate one:

sudo su - hudson
 
# cd to the home dir, default in /var/lib/hudson
cd ~
 
# create the ssh key in the ~/.ssh directory
ssh-keygen -t rsa
 
# that creates the public/private key in the /var/lib/hudson.ssh directory
cat ~/.ssh/id_rsa.pub

Copy that key to your clipboard (make sure to get all of it, it’s all one big line that probably has wrapped in your terminal window a couple of times). We’ll be using this as our identification on bitbucket.

Give the Public Key to BitBucket

If you’ve already got a BitBucket account that you want to give Hudson access to go to it now. Otherwise, create a new account for Hudson to use. Go to the “account” page and paste the public key in the “SSH Keys” field and hit the “add key” button.

If this is the account that owns the repository (or already has access to it), you’re done on BitBucket. Otherwise, if it’s a private repository owned by another user, you’ll need to add the hudson bitbucket user as a “read-only” user to the repository on the repository’s “admin” screen (or ask the repo owner to do that).

Then, go to the repository and get the ssh clone string for the repo you want to clone. By default, bitbucket shows you the HTTPS clone string, which will NOT work with this authentication method. To see it, click on the “SSH” link and it should show you something like:

hg clone ssh://hg@bitbucket.org/tednaleid/grails-test-data/

If the protocol in the url is “ssh”, you’ve got the right one.

Finish Hudson Setup

Back on the hudson box, you’ll want to manually clone the repo as the hudson user so that bitbucket is in the ~/.ssh/known_hosts file. Just execute the ssh clone command we found above:

Example:

hg clone ssh://hg@bitbucket.org/tednaleid/grails-test-data/

ssh will prompt you to add the host to the list of known hosts and then will clone the repo. If that works, authentication is set up right and we’re almost done.

Install the Mercurial plugin in Hudson

All that’s left to do now is install the Mercurial plugin in hudson. In a browser, go to http://INSERT_YOUR_IP_HERE:8080. Hudson should come up.

Click on “Manage Hudson” and go to “Manage Plugins”. Go to the “Available” tab, check “Hudson Mercurial plugin” and hit the “Install” button. Hudson will prompt you to restart, and then it’s installed.

After that, just create a new job and you’ll have a new “mercurial” option in the “source control management” section. Select that and put the ssh URL in the “Repository URL” field. Then put “default” in the “branch” field and set up the rest of the job to build/test your code (an exercise left to the reader).

The turnout at the meeting was great, and a heated discussion followed the presentation (as it always does at that meeting).

I’ve checked the original keynote files along with a PDF version into a mercurial repository.

Here’s the presentation on SlideShare:

I’ve made some enhancements to them in the interim that have made them even easier to use.

The most important alias is gtaf, which is short for “grails test-app” for failed tests.

It will search through your test output directory and look for any tests that failed. If it finds any, it will rerun only those tests. Otherwise, it will rerun all tests. That makes it easy to just use gtaf all the time. If any tests fail, it will open them up using Console.app.

If you’re not on OSX, or would like to use something else to view the failed test logs, just modify the testlog alias to do something different.

There are alternative versions of it to only run failed integration tests (gtaif – Grails Test App Integration Failed), failed unit tests (gtauf – Grails Test App Unit Failed) or attach a debugger to the tests as they run (gtadf – Grails Test App Debug Failed).

These aliases, plus most of the rest of my zsh setup, is available on bitbucket. If you haven’t used zsh before and are still using bash, I suggest switching over for the numerous benefits that it gives you.

Here’s the section of zshrc_general that has the grails testing aliases, just add this to your .zshrc/.bashrc and make sure to uncomment the appropriate GRAILS_TEST_LOG_DIRECTORY export if you’re not running grails 1.2 yet:

# grails > 1.2
export GRAILS_TEST_LOG_DIRECTORY=target/test-reports
 
# grails < 1.2
# export GRAILS_TEST_LOG_DIRECTORY=test/reports
 
# after grails-test if there were ERROR messages, you can open those logs with the Console.app using this
alias testlog='for F in `grep -lE "FAILED|Caused\ an\ ERROR" $GRAILS_TEST_LOG_DIRECTORY/plain/*.txt`; do echo ">>> opening" $F; open -a Console $F; done;'
 
# grails-debug-suspend doesn't exist by default, it's a copy of grails-debug with the suspend flag changed to "y" so that
# we can attach a remote debugger before it proceeds past startup
 
# aliases where you can optionally pass in a set of tests to run (or no argument to run all tests in that group)
alias gta=grailsTestApp
alias gtad=grailsTestAppDebug
alias gtau=grailsTestAppUnit
alias gtaud=grailsTestAppUnitDebug
alias gtai=grailsTestAppIntegration
alias gtaid=grailsTestAppIntegrationDebug
 
# aliases that will rerun any failed tests (or all tests if there aren't any failed tests)
alias gtaf=grailsTestAppFailed
alias gtadf=grailsTestAppDebugFailed
alias gtauf=grailsTestAppUnitFailed
alias gtaudf=grailsTestAppUnitDebugFailed
alias gtaif=grailsTestAppIntegrationFailed
alias gtaidf=grailsTestAppIntegrationDebugFailed
 
 
function grailsTestApp() { grailsTest grails "" $1 }
function grailsTestAppFailed() { grailsFailedTests grails "" }
 
function grailsTestAppDebug() { grailsTest grails-debug-suspend "" $1 }
function grailsTestAppDebugFailed() { grailsFailedTests grails-debug-suspend "" }
 
function grailsTestAppUnit() { grailsTest grails -unit $1 }
function grailsTestAppUnitFailed() { grailsFailedTests grails -unit }
 
function grailsTestAppUnitDebug() { grailsTest grails-debug-suspend -unit $1 }
function grailsTestAppUnitDebugFailed() { grailsFailedTests grails-debug-suspend -unit }
 
function grailsTestAppIntegration() { grailsTest grails -integration $1 }
function grailsTestAppIntegrationFailed() { grailsFailedTests grails -integration }
 
function grailsTestAppIntegrationDebug() { grailsTest grails-debug-suspend -integration $1 }
function grailsTestAppIntegrationDebugFailed() { grailsFailedTests grails-debug-suspend -integration }
 
function grailsFailedTests() {
	FAILED_TESTS=''
	for F in `grep -lE "FAILED|Caused\ an\ ERROR" $GRAILS_TEST_LOG_DIRECTORY/plain/*.txt`; do
		FAILED_TESTS="$FAILED_TESTS `echo $F | sed -E 's/.*TEST-(.*)Tests.txt/\1/g'`"
	done;
	echo "failed tests: $FAILED_TESTS"
	grailsTest $1 $2 $FAILED_TESTS
}
 
function grailsTest() {
	echo "Running: $1 test-app $2 $3 || testlog "
    time $1 test-app $2 $3 || testlog
}

I believe that they’ll also work fine as-is in bash, though I haven’t tested them.

Whenever she’d try to print, the job would say “On Hold: Authentication Required”. After some fiddling around I found a solution:

1. Go into System Preferences and go to the Print & Fax panel.
2. Right click on the printer and choose “Reset Printing System…”. This will remove all of your printers.
3. Hit the “+” and add the printer back, it should now be reset and have fixed whatever was fubar before.
4. Click on the “Sharing Preferences” button (or go to the “Sharing” System Preference panel)
5. Make sure that “Printer Sharing” is checked and that your printer’s name is also checked and that “Everyone” is set to “Can Print”
6. On the remote machine you want to remove the printer and add it back in through the System Preferences Print & Fax panel

(This is on 10.6.1, not sure if this works in 10.6.0 (where I first had the problem), you should update to the latest).

After doing all that, I was able to print from my remote computers without any problems or any issues around authentication.

I’m a big fan of storing information in markdown format as it is easy to read, easy to write, and easy to transform. A number of big websites like stackoverflow.com support the entry and display of markdown text to help people format their questions and answers without having to remember to properly format HTML.

If you’re unfamiliar with Markdown there are a number of good references on it’s use.

The grails markdown plugin is a very basic plugin. Currently, it just wraps the markdown libarary with a single TagLib that lets you easily render html from a markdown string within your gsp pages.

To install it, just type:

grails install-plugin markdown

To use it in your gsp file:

<markdown:renderHtml>
The *four* cardinal directions are
 
- North
- South
- East
- West
</markdown:renderHtml>

renders the html:

<p>The <em>four</em> cardinal directions are</p>
<ul>
    <li>North</li>
    <li>South</li>
    <li>East</li>
    <li>West</li>
</ul>

Or you can use the “text” attribute of the taglib:

<markdown:renderHtml text="${post.body}"/>

See the Grails Markdown Wiki and source for more details.

A number of interesting techniques have popped out of this that I think would be useful for others doing similar types of importing using GORM/Hibernate/MySQL.

Background

For background, the imports in our app happen a few times a week, and range in size from less than a hundred up to a few hundred thousand new records.

It is also possible for us to get updates on previous data that we’ve imported, if a record in the import matches a record in our database, we want to update the existing record rather than inserting a duplicate.

We had a desire to use a Grails service to do the importing as we could leverage our existing data model, plus, there are some special cases where the import logic can get a bit hairy and having access to all of our domain methods is nice.

If you’re doing much more frequent imports, with heavy performance requirements using millions to hundreds of millions of rows, you’ll likely want to look at another tool more specifically targeted for batch/ETL processes or writing something directly in Java. For smaller jobs, you might be surprised how far you can get with groovy and grails.

Example Naive Import Application

To demonstrate the techniques, I’ve created a small test application. This simple app only has Books. It imports 100,000 randomly generated books in a single job (importing a Library of Books). If our database already has a book with that ISBN and edition number, we want to update the title of previous book rather than creating a new book.

Here are the setup files that won’t change as we tweak our application.

The Domain class:

// domain/Book.groovy
class Book {
    String title
    String isbn
    Integer edition
}

The BookController that calls the service:

// controller/BookController.groovy
class BookController {
 
    def bookService
 
    def index = { 
        def library = new Library()
        def startTime = System.currentTimeMillis()		
        bookService.importBooksInLibrary(library)	
        render "time: ${(startTime - System.currentTimeMillis())/1000} seconds"
    }
}

And a Library class that’s the source of our books (don’t worry too much about the details of the Library class, just know that it’s the source of the fields that we’ll be importing. For each book, it simply returns a map of the field values that should be saved or updated for the Book):

class Library implements Iterator {
    def startTime = System.currentTimeMillis()
    def lastBatchStarted = startTime
    def BOOKS_IN_LIBRARY = 100000
    def currentBook = 1
    def random = new Random(12345) // random generator with defined seed
 
    Iterator iterator() { return this as Iterator }
    boolean hasNext() { currentBook <= BOOKS_IN_LIBRARY }
    void remove() { }
    def next() {
        if (! (currentBook % 100) ) printStatus()
 
        return [
            title: "Book ${currentBook++}",
            isbn: randomIsbn(), 
            edition: randomEdition()
        ]
    }
 
    def randomIsbn() {
        // one of 50,000 random isbn numbers
        return "isbn-${random.nextInt(50000)}"
    }
 
    def randomEdition() {
        // first through tenth editions
        return random.nextInt(9) + 1
    }
 
    def printStatus() {
        def batchEnded = System.currentTimeMillis()
        def seconds = (batchEnded-lastBatchStarted)/1000
        def total = (batchEnded-startTime)/1000
        println "Library Book $currentBook, last batch: ${seconds}s, total: ${total}s"
        lastBatchStarted = batchEnded
    }
}

The Library class is a proxy for whatever the source of your import info is (a .csv/excel file that you’re parsing through, a webservice, something posted to your server, etc.).

Also, to help see how we affect timing, every 100 books the Library prints out how long it took to process the last 100 books.

Here is the naive BookService that iterates through the Library that the BookController gives it. For each book it receives a map of the books values. It does a findByIsbnAndEdition() to see if the book already exists. If it does, it updates it, otherwise it will create a new book:

class BookService {
    def importBooksInLibrary(library) {		
        library.each { Map bookValueMap ->
            updateOrInsertBook(bookValueMap)
        }		
    }
 
    def updateOrInsertBook(Map bookValueMap) {
        def title = bookValueMap.title
        def isbn = bookValueMap.isbn
        def edition = bookValueMap.edition
        def existingBook = Book.findByIsbnAndEdition(isbn, edition)
 
        if (existingBook) { // just update title
            existingBook.title = title
            existingBook.save()
        } else { 
            new Book(bookValueMap).save()
        }
    }
}

When you run this code, the first thing you notice is how slow it is. The first 1000 records aren’t that bad. On my old laptop they take about 7 seconds to insert. After that things get slower and slower, books 10,000 to 11,000 take 20 seconds to insert. Importing gets progressively worse as time goes on.

Yikes! It takes 2 hours and 36 minutes to import 100k records? Over 3 minutes for the last thousand books to get imported? That can’t be right.

As it runs, if you open up your activity monitor/task manager/top application, you’ll also see that Grails is running very hot but also that MySQL’s CPU usage is slowly going up over time.

MySQL Database Performance Tweaks

Start with the obvious: Database Indexes

If you’ve done much database work, you know how much indexes can improve the speed of a query. But how do you know what queries need indexes?

In some situations, like our example above, it can be fairly obvious. We’re calling Book.findByIsbnAndEdition(). Chances are we need an index on the “book” table that includes the “isbn” and “edition” fields:

CREATE INDEX `idx_book_isbn_edition` ON `book`(`isbn`, `edition`);

What if it isn’t so obvious which queries need an index?

There are a number of techniques available to us through grails to see what kind of SQL is getting executed.

Turn on grails SQL logging

Within grails, you can turn on SQL logging by putting this in your Config.groovy’s log4j closure:

trace 'org.hibernate.SQL'

Install the p6spy plugin and JDBC SQL profiler

The p6spy grails plugin inserts a proxy between your database driver classes and Grails. Because of this, it knows the full query, including the filled in query parameters and some timing information.

This is useful by itself, but you can have the p6spy plugin log it’s messages to a log4j socket appender that talks to the JDBC SQL Profiler which will aggregate all of the information and help you decide where you’ve got slow queries and need indexes.

Mike Hugo has a great post on using the p6spy plugin with the JDBC SQL profiler.

Turn on the MySQL Slow Query Log

If you’re using a recent version of MySQL, there’s an easy, built-in way to see which queries are slow and which queries aren’t using indexes: the slow query log.

To turn it on, you simply need to add a few lines to your mysql cnf file (by default on OSX just create a file at /etc/my.cnf and it will get loaded):

[mysqld]
log-slow-queries
#Log all queries > long_query_time seconds, default 10 seconds
long_query_time=1

This will automatically create a log file in your mysql “data” directory called host_name-slow.log that contains all queries taking longer than 1 second.

You can optionally add

log-queries-not-using-indexes

And this will also log those queries that don’t have an index (but don’t take longer than 1 second to execute).

MySQL also comes with a perl script to parse through the slow log file and show you the number of times each query is slow with some additional statistics. To run it, you need to add basedir to your my.cnf file. The basedir value just holds the location of mysql’s data directory.

basedir=/usr/local/mysql

Then run the script:

mysqldumpslow

These steps should get you a long way towards figuring out which tables need indexes.

Turn on the MySQL General Query log

MySQL has an additional query log called the general query log. It logs everything that comes into MySQL and is extremely noisy. One benefit of it though is that it gives you the full query with all parameters, so it’s more useful than turning on grails SQL trace log and potentially a little quicker to set up than the p6spy plugin.

You can enable it by putting this in your my.cnf:

general_log=1

Suggested Reading for MySQL indexes

If you haven’t created many indexes before, I highly suggest reading this post on indexes on the mysql high performance blog. The executive summary is that multi-key indexes tailored to your query have a performance edge over individual key indexes on each of the fields. If you’re executing a query enough times, it might be worth creating a multi-key index like we have above.

MySQL Database Engines

If MySQL is still your performance bottleneck, or if you’re finding that everything else comes to a stop when you’re running your batch import, check which storage engine you’re using.

For grails applications, I’d prefer InnoDB (or Falcon) to the default MyISAM. MyISAM is good for read-only situations where you don’t care about enforced foreign key releationships. Switching to InnoDB has significantly increased the performance of my app in a number of situations. InnoDB also supports row level locking, whereas MyISAM will lock the entire table when doing an update or an insert.

If you haven’t explicitly done anything, chances are that all of your grails-generated tables are MyISAM. You can change the default storage engine for mysql by specifying a different engine in your my.cnf:

default-storage-engine=innodb

To see the default is for your database execute:

SHOW engines;

To see the engine for a particular table, use “show create table”:

mysql> SHOW CREATE TABLE book\G
*************************** 1. row ***************************
       TABLE: book
CREATE TABLE: CREATE TABLE `book` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `version` bigint(20) NOT NULL,
  `edition` int(11) NOT NULL,
  `isbn` varchar(255) NOT NULL,
  `title` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=84192 DEFAULT CHARSET=latin1
1 row IN SET (0.00 sec)

If you need to alter the storage engine for an existing table, you can execute the alter table SQL statement:

ALTER TABLE book ENGINE = INNODB;

Grails Performance Tweaks

Adding an index and modifying the storage engine of our example book table to InnoDB make our sample app 25-30% faster. A nice boost from where we were originally, but things are still way too slow:

Adding indexes to the database drops the database CPU utilization to almost nothing, but grails is still pegged and it still gets much slower over time.

Something is amiss.

At this point, some people might be inclined to believe the myth that “grails is slow” and go through a painful rewrite in another “faster” language/technology.

Don’t go there yet! It’s only slow if you don’t understand what’s actually going on under the covers.

After reading through a couple of very helpful posts by Burt Beckwith, and reading through the Hibernate chapter on batch processing, we have all of the information we need to speed things up significantly.

Use a Monitoring Tool to Watch Memory/CPU usage Over Time

Because the import gets significantly slower over time, even with the index applied, we know there is some sort of leak.

If you don’t have your code instrumented with timing logic already in it, it’s easy to pop open jconsole or jvisualvm (both already installed and likely in your classpath if you have Java 1.6), connect to your grails instance and watch performance/cpu utilization over time.

The Culprits: Hibernate First-Level Cache and Grails Validation Cache

I found on Burt Beckwith’s blog, that there are 2 separate leaks, one of them is in the hibernate first-level cache, the other is a map that Grails uses for domain object validation errors.

Normally, a grails hibernate session executes something quickly and returns. During importing, we do a ton of processing, all with the same hibernate session. All of these objects that would normally be garbage collected when the session closed are piling up.

The easiest way to deal with this is to create a simple method to clear out these collections periodically.

We can modify our BookService to clean up GORM after every 100 books we insert:

class BookService {
 
    def sessionFactory
    def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP
 
    def importBooksInLibrary(library) {
        library.eachWithIndex { Map bookValueMap, index ->
            updateOrInsertBook(bookValueMap)
            if (index % 100 == 0) cleanUpGorm()
        }
    }    
 
    def cleanUpGorm() {
        def session = sessionFactory.currentSession
        session.flush()
        session.clear()
        propertyInstanceMap.get().clear()
    }
 
    def updateOrInsertBook(Map bookValueMap) {
        // ... same as above
    }
}

This change makes everything much better. All 100,000 books get inserted in 3 minutes, the same time that it took the naive example to insert the last 1,000 books.

Also, because MySQL is now properly using indexes, the load on it is very light. If we’re looking for even more speed improvement, this would allow us to split our work up into batches and to make multiple BookService calls (potentially on different web servers). Because we’ve switched to using InnoDB, we’re now working with row-level locking, which isn’t hostile to this approach unlike MyISAM and it’s full table locking on Insert/Update.

Conclusion

This post is just some of what can be done to improve batch performance with Grails and MySQL. Using these tips, I was able to speed up my own imports to a level that met the speed requirments for our problem space, but kept our code much more maintainable. Our domain object remain the canonical code representation of our database model.

There are a number of other things that could be looked at to further speed up batch performance and that could be important to your particular application/infrastructure.

• connectivity between app server and DB
• giving enough memory to your app and DB servers
• other load into database and app servers
• anemic hardware
• poor database design

Be sure to weigh the cost of making changes in each of these areas, and run benchmark tests before committing to them. You’ll often find that what you think is the bottleneck, isn’t. Speed is a feature, but you need to balance it with the costs to achieve it.

Let me know if you see any issues or broken links.