I write my own custom iterators all the time and name them something descriptive. This makes the code much more readable. Rather than trying to decipher what a for loop is trying to do, we wrap up all of that iteration logic into a meaningful name and we cleanly separate that iteration from the processing that we’re doing with each element.

This kind of design is a core concept in Uncle Bob’s Clean Code, one of my favorite programming books in the last few years.

This example iterates over a collection and calls the passed in closure until we hit a value greater than 5.

def eachUntilGreaterThanFive = { collection, closure ->
    for ( value in collection ) {
        if ( value  > 5 ) break
        closure(value)
    }
}
 
def a = [1, 2, 3, 4, 5, 6, 7]
 
eachUntilGreaterThanFive(a) {
    println it
}

prints:

1
2
3
4
5

This code makes it obvious what the iterator is doing (looping till we hit a condition) as well as what will happen with each element iterated over (print it out).

For a real life example, I had a need to iterate over a list of values and where I needed both the current object as well as a peek at the next object in the list.

Doing this is Java is a bit of a pain, but groovy makes it easy to write and (hopefully) to read, we can also add it directly onto the Collection metaClass so that it’s available for all of our Collection instances:

Collection.metaClass.eachWithPeek = { closure ->
    def last = null
    delegate?.each { current ->
        if (last) closure(last, current)
        last = current
    }
    if (last) closure(last, null)
}

These test cases show that as we iterate through the collection, we can see the current item and peek at the next one (if any). If the collection is empty, we don’t execute the closure it, and if we’re at the end of the list there isn’t anything to peek at:

[].eachWithPeek { current, peek ->
    assert false // shouldn't get here, nothing to iterate through
}
 
[1].eachWithPeek { current, peek ->
    assert current == 1
    assert peek == null  // only 1 element, nothing to peek at
}
 
def results = []
[1, 2, 3, 4, 5].eachWithPeek { current, peek ->
    results << [current, peek]
}
assert results == [[1, 2], [2, 3], [3, 4], [4, 5], [5, null]]

By default, grails uses ~/.grails as the working directory. If you’re doing branchy development, you can run into problems with this if you’ve got plugins installed in one branch that aren’t in the other. Having a unique directory per branch prevents you from having to run grails clean all the time.

Here’s a quick shell script that changes the grails working directory to have the branch name as a suffix if your source is contained in a mercurial repository (ex: the default branch would be ~/.grails_default and the 1.0 branch would be ~/.grails_1.0). If your application is not in a repo, it just uses the regular ~/.grails directory.

#!/bin/sh
HG_BRANCH=`hg branch 2>/dev/null`
GRAILS_SCRIPT=$GRAILS_HOME/bin/grails
 
if [ $HG_BRANCH ]; then
	GRAILS_WORK_DIR=`echo ~`/.grails_$HG_BRANCH
	echo "** grails working directory: $GRAILS_WORK_DIR"
	$GRAILS_SCRIPT -Dgrails.work.dir=$GRAILS_WORK_DIR $@
else
	echo "** default grails working directory"
	$GRAILS_SCRIPT $@
fi

Just save this script as “grails” and put it in your PATH before the $GRAILS_HOME/bin directory (also make sure that you’ve defined $GRAILS_HOME). I have a ~/bin directory that’s the first thing in my PATH.

If you use the grails-debug command, you can repeat these steps for that, just change GRAILS_SCRIPT to $GRAILS_HOME/bin/grails-debug.

This same technique could easily be modified to be used for other source control systems such as git or subversion.

It adds a new buildLazy method that will only create a new object graph if it can’t find an existing object that matches the build criteria specified.

Example:

    def a = new Author(firstName: "Ray", lastName: "Bradbury")
    a.save()
    assert 1 == Author.count()
 
    // Author table already has Ray Bradbury in it, finds existing record
    def bradbury = Author.buildLazy(lastName: "Bradbury")  
    assert 1 == Author.count()
    assert bradbury.firstName == "Ray"
 
    def newAuthor = Author.buildLazy(lastName: "Moore")
    assert 2 == Author.count() // creates a new record, no existing "Moore" author previously

I find this particularly useful for building testing data in the grails console. It allows me to do some quick setup of new objects and then concentrate on creating the code that I actually want to try out. Without buildLazy, I was always commenting out my creation code so that multiple executions of the test script didn’t create multiple copies of the same data (and potentially hit unique constraints issues that I had to screw around with).

Now, I no longer need to worry about commenting out the creation code after running it the first time.

Example:

def u = User.buildLazy(ssn: "123-456-7890") // User.ssn has unique constraint
u.foo()
u.bar()
u.baz()

Before buildLazy I’d need to comment out the user creation and replace it with a user.findBySsn. Now buildLazy does the creation and later finding for me. I can concentrate on making foo/bar/baz do what I want them to (the reason I opened the console in the first place).

If you’re not familiar with build-test-data, the quick summary is that it puts a build() method on all grails domain objects. Calling that method will automatically construct and save an instance of that domain object that conforms to all of the domain’s constraints. It also allows you to override the values that you want to explicitly set. It makes your tests much cleaner and less fragile as you only need to specify the values that actually matter to a particular test method instead of building a huge graph of objects just to satisfy constraints.

The plugin has been quite stable for the past 6 months or so, and has survived upgrades from grails 1.0.X through grails 1.2.1 with minimal changes. Because of this, I’ve decided to move the version from 0.2.3 to 1.0 to indicate that the plugin is stable and ready to be used. I’ve also added the LICENSE file releasing the plugin under the Apache 2.0 open source license (the same license as grails). It was always open source, but I had neglected to add the official license file in the past.

If you’re not familiar with the build-test-data plugin, the documentation on the wiki is thorough, I’ve also given a presentation on build-test-data that explains why it’s better than other existing data generation technologies.

The biggest changes for this release compared to the last are a number of bugfixes around making sure that both sides of a one-to-many relationship get populated correctly and that there isn’t a need to refresh() from the database.

Previously, if you had an Author that hasMany Books, and each Book belongsTo an Author, you’d need to refresh the Author if you tried to build a new book with an existing author:

Author eap = Author.findByName("Edgar Allan Poe")
Book b = Book.build(author: eap, title: "The Tell-Tale Heart")
 
assertEquals eap.name, b.author.name  // works, linked in OK previously
assertEquals 1, eap.books.size() // FAILED previously WORKS now, previously the book wasn't added properly to the author side of things

The previous workaround was to call eap.refresh() to reload the author from the database, or to have the user manually addToBooks(b). Both solutions were ugly and kludgy and this issue has now been fixed.

Since then, I’ve completely abandoned bash, in favor of zsh (which I consider to be the superior shell) and I’d been getting annoyed at a few issues in the last grails autocomplete script.

I finally got motivated to make some enhancements to it. Including support for grails 1.2 plugin scripts (1.2 moved the plugins into the ~/.grails directory), and support for test class name autocompletion (very useful for grails test-app).

To get it working (assuming you’re using zsh), you can either add the contents of my zshrc_grails_compinstall to your .zshrc file, or you can switch over to using my full zshrc setup, which has a number of nice features that I’ve collected over the years. I’ll also continue to update this as I think of new tricks.

After getting it installed, just type “grails” followed by a space and then hit tab. It will show you a list of all of the potential grails scripts for the application that you’re in. It’s aware of the version of the current app, as well as it’s application name based on the contents of application.properties, and also will include any scripts for the plugins you have installed in that app in ~/.grails/GRAILS_VERSION/projects/APP_NAME/plugins (in addition to the scripts in ./scripts, $GRAILS_HOME/scripts and ~/.grails/scripts).

After you choose your script (such as grails test-app hit space again and it will show you all of the test classes, with the full package for the class, under your test directory.

One of these changes is that each sheet can actually have multiple tables on it, and these can be arranged independently on the same page. This prevents the problem that happens in excel where you have multiple sets of data you want to see at the same time, but the cell/row sizing for one set of data affects the data in the other set that happens to be on the same row.

I’ve been leveraging this for a one-off personal project and I had a need to sum up all of the data in the 2nd column on all of the tables within a particular sheet.

This brought me to AppleScript, Apple’s scripting language that’s used to drive applications.

You can write scripts in the AppleScript editor (found in /Applications/Utilities). In there, you can look up the commands and objects that a particular program makes available by going to File->Open Dictionary and then finding the application that you want to script.

I’ve done a little work in AppleScript in the past, but always recoil from it when I’m done. It’s so damn wordy in what appears to be an attempt to make it more “friendly” when it really just makes it harder to remember the right syntax to get something done. It probably just has “too many notes”.

Regardless, here’s the script that I came up with to sum up all of the values in the 2nd column (Column B) for all of the tables on the currently active sheet and then insert the result onto the 2nd column of a table named “Totals” (which must previously exist).

tell application "Numbers"
	tell document 1
		set total to 0
		set totalTable to 0
		tell sheet 1
			repeat with j from 1 to (count of tables)
				try
					tell table j
						if (name is not equal to "Totals") then
							repeat with k from 1 to count of rows
								set total to total + (value of cell 2 of row k)
							end repeat
						else
							set totalTable to j
						end if
					end tell
				end try
			end repeat
			tell table totalTable
				display dialog "total = " & total & " total table = " & totalTable
				set value of cell 2 of row 1 to total
			end tell
		end tell
	end tell
end tell

I needed this ability as the number of tables on this sheet will change quite frequently depending on what I’m doing with it, and the values in the 2nd column of each of these tables will also get frequently modified and I need to easily know the sum across all of the tables.

To actually execute the script within Numbers.app, there are a couple of ways to do it. It doesn’t appear that Numbers has it’s own script directory (normally, it would be under ~/Library/Application Support/iWork/Numbers/Scripts), but you can add it to your own User Scripts directory at ~/Library/Scripts.

If you want to access it through your menus, you can enable the global script icon up there by going into the AppleScript Editor preferences. On the General tab, check the box that says “Show Script menu in menu bar”.

If you’re using LaunchBar or Quicksilver, you can also add this directory to their catalog as a place to search. Then it’s as easy as cmd-space and the name of the script to execute it (both apps also allow you to set up keyboard shortcuts too).

In all, it feels hacky, and I’m sure it could be improved if I actually knew what the hell I was doing in AppleScript, but as I said, it’s a one-off that I need for a fairly particular problem.

I should probably take a little more time to learn more about AppleScript (or look more closely at a bridge to a language that I find a bit more palatable). Being able to script the applications that I use can be very useful.

class ClosureInterrogator {
    private Map closureValueMap = [:]
 
    static Map extractValuesFromClosure(Closure closure) {
        def interrogator = new ClosureInterrogator(closure)
        return interrogator.closureValueMap
    }
 
    private ClosureInterrogator(Closure closure) {
        def oldResolveStrategy = closure.getResolveStrategy()
        def oldDelegate = closure.getDelegate()
        closure.delegate = this
        closure.resolveStrategy = Closure.DELEGATE_FIRST
 
        try {
            closure()
        } finally {        
            closure.setDelegate(oldDelegate)
            closure.setResolveStrategy(oldResolveStrategy)
        }
    }
 
    // property getter
    def propertyMissing(String name) {
        return closureValueMap[name]
    }
 
    // property setter
    def propertyMissing(String name, value) {
        closureValueMap[name] = value
    }
 
    def methodMissing(String name, args) {
        if (args.size() == 1) {
            closureValueMap[name] = args[0]
        } else {
            closureValueMap[name] = args
        }
    }
}

This technique is the basis for all groovy mocking techniques and libraries, it’s also the main technique used to create a DSL with groovy.

Here is a sample class that has a closure (“something”) that we want to extract the values from.

class SomeClass {
    static something = {
        key1 "value1"              // calls methodMissing("key1", ["value1"])
        key2("value2")             // calls methodMissing("key2", ["value2"])
        key3 = "value3"            // calls propertyMissing("key3", "value3")
        key4 "foo", "bar", "baz"   // calls methodMissing("key4", ["foo","bar","baz"])
    }
}

If we call “extractValuesFromClosure” on the “something” closure, we’ll get back a map that has all of the values we want in it.

def closureValueMap = ClosureInterrogator.extractValuesFromClosure(new SomeClass().something)
 
assert "value1" == closureValueMap."key1"  // calls propertyMissing("key1")
assert "value2" == closureValueMap."key2"  // calls propertyMissing("key2")
assert "value3" == closureValueMap."key3"  // calls propertyMissing("key3")
assert ["foo", "bar", "baz"] == closureValueMap."key4"  // calls propertyMissing("key4")

This class will work with any type of closure, such as this closure used by the grails mail plugin to send mail.

def mailClosure = {
   to "fred@g2one.com","ginger@g2one.com"
   from "john@g2one.com"
   cc "marge@g2one.com", "ed@g2one.com"
   bcc "joe@g2one.com"
   subject "Hello John"
   body 'this is some text'
}
 
def results = ClosureInterrogator.extractValuesFromClosure(mailClosure)
println results
// prints:
// [to:[fred@g2one.com, ginger@g2one.com], from:john@g2one.com, cc:[marge@g2one.com, ed@g2one.com], bcc:joe@g2one.com, subject:Hello John, body:this is some text]
 
assert results.to.every { it.endsWith("g2one.com") }

If we wanted to assert that within a test environment, we’re only sending e-mail to a specific domain, this would be one technique to assert it before we actually send the e-mail off.

ClosureInterrogator is generic enough that it could also be used on a number of Grails/GORM domain closures such as “constraints” and “mapping”.

I responded with a long reply that I thought would be worth cleaning up and posting here.

Mercurial, Bitbucket and Hudson

I’ve been using Mercurial as my personal revision control system for a while now.

I’ve more recently been using it at work with a repository hosted at BitBucket, a great service that offers free accounts (including one private repo) and very reasonable upgraded accounts. (I don’t have any relationship with bitbucket other than just as a happy and satisfied user.)

Being a good agile shop, we use hudson as our continuous integration server (much better than Cruise Control IMO). It polls the source control repository on a periodic basis, if it sees any code changes, it checks them out and runs all of the tests.

Quick Mercurial/Hudson Setup

Install Mercurial

There are lots of ways to install mercurial. On the Ubuntu box, the quickest way to get a recent version for me was to just use python’s easy_install:

sudo easy_install mercurial

Install Hudson

There are better guides out there for getting Hudson going depending on the platform you’re using, but here’s the quick instructions for how I got it going on an Ubuntu 9.10 image running on Amazon EC2.

Using Ubuntu’s apt-get package management, installing hudson is easy. You need to add the repository that has the hudson source so that apt-get knows where to get it. Add this to your /etc/apt/sources.list file:

deb http://hudson-ci.org/debian binary/

Then update apt-get and install hudson:

sudo apt-get update
sudo apt-get install hudson

That creates a hudson install with a new “hudson” user on your system.

Setting up Hudson to SSH

We want this user to be able to connect to BitBucket over SSH and check out the repository. To do this, we’ll need an ssh public and private key that hudson can use. Sudo to the hudson user and generate one:

sudo su - hudson
 
# cd to the home dir, default in /var/lib/hudson
cd ~
 
# create the ssh key in the ~/.ssh directory
ssh-keygen -t rsa
 
# that creates the public/private key in the /var/lib/hudson.ssh directory
cat ~/.ssh/id_rsa.pub

Copy that key to your clipboard (make sure to get all of it, it’s all one big line that probably has wrapped in your terminal window a couple of times). We’ll be using this as our identification on bitbucket.

Give the Public Key to BitBucket

If you’ve already got a BitBucket account that you want to give Hudson access to go to it now. Otherwise, create a new account for Hudson to use. Go to the “account” page and paste the public key in the “SSH Keys” field and hit the “add key” button.

If this is the account that owns the repository (or already has access to it), you’re done on BitBucket. Otherwise, if it’s a private repository owned by another user, you’ll need to add the hudson bitbucket user as a “read-only” user to the repository on the repository’s “admin” screen (or ask the repo owner to do that).

Then, go to the repository and get the ssh clone string for the repo you want to clone. By default, bitbucket shows you the HTTPS clone string, which will NOT work with this authentication method. To see it, click on the “SSH” link and it should show you something like:

hg clone ssh://hg@bitbucket.org/tednaleid/grails-test-data/

If the protocol in the url is “ssh”, you’ve got the right one.

Finish Hudson Setup

Back on the hudson box, you’ll want to manually clone the repo as the hudson user so that bitbucket is in the ~/.ssh/known_hosts file. Just execute the ssh clone command we found above:

Example:

hg clone ssh://hg@bitbucket.org/tednaleid/grails-test-data/

ssh will prompt you to add the host to the list of known hosts and then will clone the repo. If that works, authentication is set up right and we’re almost done.

Install the Mercurial plugin in Hudson

All that’s left to do now is install the Mercurial plugin in hudson. In a browser, go to http://INSERT_YOUR_IP_HERE:8080. Hudson should come up.

Click on “Manage Hudson” and go to “Manage Plugins”. Go to the “Available” tab, check “Hudson Mercurial plugin” and hit the “Install” button. Hudson will prompt you to restart, and then it’s installed.

After that, just create a new job and you’ll have a new “mercurial” option in the “source control management” section. Select that and put the ssh URL in the “Repository URL” field. Then put “default” in the “branch” field and set up the rest of the job to build/test your code (an exercise left to the reader).

The turnout at the meeting was great, and a heated discussion followed the presentation (as it always does at that meeting).

I’ve checked the original keynote files along with a PDF version into a mercurial repository.

Here’s the presentation on SlideShare:

I’ve made some enhancements to them in the interim that have made them even easier to use.

The most important alias is gtaf, which is short for “grails test-app” for failed tests.

It will search through your test output directory and look for any tests that failed. If it finds any, it will rerun only those tests. Otherwise, it will rerun all tests. That makes it easy to just use gtaf all the time. If any tests fail, it will open them up using Console.app.

If you’re not on OSX, or would like to use something else to view the failed test logs, just modify the testlog alias to do something different.

There are alternative versions of it to only run failed integration tests (gtaif – Grails Test App Integration Failed), failed unit tests (gtauf – Grails Test App Unit Failed) or attach a debugger to the tests as they run (gtadf – Grails Test App Debug Failed).

These aliases, plus most of the rest of my zsh setup, is available on bitbucket. If you haven’t used zsh before and are still using bash, I suggest switching over for the numerous benefits that it gives you.

Here’s the section of zshrc_general that has the grails testing aliases, just add this to your .zshrc/.bashrc and make sure to uncomment the appropriate GRAILS_TEST_LOG_DIRECTORY export if you’re not running grails 1.2 yet:

# grails > 1.2
export GRAILS_TEST_LOG_DIRECTORY=target/test-reports
 
# grails < 1.2
# export GRAILS_TEST_LOG_DIRECTORY=test/reports
 
# after grails-test if there were ERROR messages, you can open those logs with the Console.app using this
alias testlog='for F in `grep -lE "FAILED|Caused\ an\ ERROR" $GRAILS_TEST_LOG_DIRECTORY/plain/*.txt`; do echo ">>> opening" $F; open -a Console $F; done;'
 
# grails-debug-suspend doesn't exist by default, it's a copy of grails-debug with the suspend flag changed to "y" so that
# we can attach a remote debugger before it proceeds past startup
 
# aliases where you can optionally pass in a set of tests to run (or no argument to run all tests in that group)
alias gta=grailsTestApp
alias gtad=grailsTestAppDebug
alias gtau=grailsTestAppUnit
alias gtaud=grailsTestAppUnitDebug
alias gtai=grailsTestAppIntegration
alias gtaid=grailsTestAppIntegrationDebug
 
# aliases that will rerun any failed tests (or all tests if there aren't any failed tests)
alias gtaf=grailsTestAppFailed
alias gtadf=grailsTestAppDebugFailed
alias gtauf=grailsTestAppUnitFailed
alias gtaudf=grailsTestAppUnitDebugFailed
alias gtaif=grailsTestAppIntegrationFailed
alias gtaidf=grailsTestAppIntegrationDebugFailed
 
 
function grailsTestApp() { grailsTest grails "" $1 }
function grailsTestAppFailed() { grailsFailedTests grails "" }
 
function grailsTestAppDebug() { grailsTest grails-debug-suspend "" $1 }
function grailsTestAppDebugFailed() { grailsFailedTests grails-debug-suspend "" }
 
function grailsTestAppUnit() { grailsTest grails -unit $1 }
function grailsTestAppUnitFailed() { grailsFailedTests grails -unit }
 
function grailsTestAppUnitDebug() { grailsTest grails-debug-suspend -unit $1 }
function grailsTestAppUnitDebugFailed() { grailsFailedTests grails-debug-suspend -unit }
 
function grailsTestAppIntegration() { grailsTest grails -integration $1 }
function grailsTestAppIntegrationFailed() { grailsFailedTests grails -integration }
 
function grailsTestAppIntegrationDebug() { grailsTest grails-debug-suspend -integration $1 }
function grailsTestAppIntegrationDebugFailed() { grailsFailedTests grails-debug-suspend -integration }
 
function grailsFailedTests() {
	FAILED_TESTS=''
	for F in `grep -lE "FAILED|Caused\ an\ ERROR" $GRAILS_TEST_LOG_DIRECTORY/plain/*.txt`; do
		FAILED_TESTS="$FAILED_TESTS `echo $F | sed -E 's/.*TEST-(.*)Tests.txt/\1/g'`"
	done;
	echo "failed tests: $FAILED_TESTS"
	grailsTest $1 $2 $FAILED_TESTS
}
 
function grailsTest() {
	echo "Running: $1 test-app $2 $3 || testlog "
    time $1 test-app $2 $3 || testlog
}

I believe that they’ll also work fine as-is in bash, though I haven’t tested them.