Introduction to Using Redis With Groovy

| Comments

I’m more excited about Redis than just about any other technology right now.

Redis is an insanely fast key/value store, in some ways similar to memcached, but the values it stores aren’t just dumb blobs of data, but can also be hashes, lists, sets and sorted sets. It provides a number of atomic operations on each of those data types (ex: union and intersection methods on sets) and it has been called a “data structure server”.

It’s used in production today by a number of very popular websites including Craigslist, GitHub, The Guardian, and Digg.

Redis is of particular note for Groovy/Grails developers because it’s development is financially supported by VMWare/SpringSource. Redis was also the first non-relational data store to be officially supported by core Grails developers.

It has excellent documentation and a super simple wire protocol that has made it easy for a ton of client libraries for just about every language to pop up. You could probably write a simple client library in a day. Once you know the commands, you can even interact with it through telnet (though you don’t have to :).

Installing Redis

There are a number of ways to install redis. For testing, I like to grab the source and play with that:

% git clone git://github.com/antirez/redis.git
% make

Just in case, you might want to run the tests to ensure the master branch you just grabbed is working (they take about 5 minutes to run).

% make test

Starting up a server is as easy as

% src/redis-server

You can then benchmark it on your system to see the performance:

% src/redis-benchmark 
====== PING ======
  10000 requests completed in 0.19 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

70.03% <= 1 milliseconds
100.00% <= 1 milliseconds
51546.39 requests per second
...

By default, it runs on port 6379 though you can override defaults with a configuration file.

It has a simple REPL built in where you can issue commands and see the results.

% src/redis-cli
redis> set tick "SPOOOON!!!"
OK
redis> get tick
"SPOOOON!!!"

You can use the monitor command to watch all of the requests that redis processes. If we had a monitoring session running while issuing the commands above, this is what it would look like:

% src/redis-cli
redis> monitor
OK
1293518654.694160 "monitor"
1293518700.558202 "set" "tick" "SPOOOON!!!"
1293518702.69875 "get" "tick"

Using Groovy’s Grape Annotation to Grab the Jedis Java client library

The jedis java library provides a thin layer over the wire protocol. Redis Commands are simply turned into method names so the redis documentation also works well as jedis documentation.

If you’re using a recent version of groovy, you can leverage the grape dependency system to automatically download the jedis jar and include it in our classpath. Connecting to our Redis localhost server running on the default port (6379) is as simple as this:

#!/usr/bin/env groovy
@Grapes([
    @Grab('redis.clients:jedis:1.5.1'),
    @GrabConfig(systemClassLoader=true)
])

import redis.clients.jedis.*

Jedis jedis = new Jedis("localhost")

Strings

Strings are the simplest data type in Redis. We can use set to save them and get to retrieve a saved value.

jedis.set("foo", "bar")
assert jedis.get("foo") == "bar"

We can add a little groovy meta-programming to make working with simple strings (GET/SET) a little easier through property accessors.

// let's make Jedis a little more groovy and call get/set through property accessors
Jedis.metaClass.getProperty = { String name ->
    delegate.get(name)
}

Jedis.metaClass.setProperty = { String name, value ->
    delegate.set(name, value)
}

Jedis jedis = new Jedis("localhost")

// get/set basic String values
jedis.name = "Ted Naleid"

assert jedis.name == "Ted Naleid"

Integers/Counters

Redis stores all numeric String values internally as integers that can be atomically incremented and decremented. If the value doesn’t exist, incrementing it sets it to 1.

jedis.beanCounter = "4"
assert jedis.incr("beanCounter") == 5
assert jedis.incr("beanCounter") == 6
jedis.del("beanCounter")
assert jedis.incr("beanCounter") == 1

If the value we’re incrementing isn’t an integer, redis it’ll throw an execption when we try to increment it.

jedis.notAnInteger = "foo"

try {
    jedis.incr("notAnInteger")
    throw new Exception("shouldn't get here, need an integer to increment")
} catch (redis.clients.jedis.JedisException e) {
    assert e.message == "ERR value is not an integer or out of range"
}

Expire and TTL (Time To Live)

Redis has a number of useful generic functions including the ability to act as a cache. Just tell Redis when the key should expire (in seconds).

jedis."expire:me" = "value to expire"
jedis.expire("expire:me", 60)
assert jedis.ttl("expire:me") > 50 

// -1 means "never expire"
assert jedis.ttl("brand new value") == -1

Lists

The list data structure works very much like it does in groovy/java.

You can push items onto the end of the list:

// if the list doesn't exist, it'll create it and add the initial element
jedis.rpush("alphabet", "Bravo")
jedis.rpush("alphabet", "Charlie")

Or the start of the list:

jedis.lpush("alphabet", "Alpha")

To see the contents of a list, you need to ask it for a range of elements. lrange shows a slice of a 0-based list. The last parameter can use negative indexes to refer to an offset from the end of the list.

-1 means the last element, -2 is the 2nd to last element, etc.

assert jedis.lrange("alphabet", 0, -1) == ["Alpha", "Bravo", "Charlie"] // whole list
assert jedis.lrange("alphabet", 0, -2) == ["Alpha", "Bravo"] // all but last item
assert jedis.lrange("alphabet", 0,  1) == ["Alpha", "Bravo"]  // first and 2nd items
assert jedis.lrange("alphabet", 1,  2) == ["Bravo", "Charlie"] // 2nd and 3rd items

assert jedis.llen("alphabet") == 3

You can also pop elements off of the start or the end of the list.

// pop the first item off the list
assert jedis.lpop("alphabet") == "Alpha"

// pop the last item off the list
assert jedis.rpop("alphabet") == "Charlie"

// only one thing left
assert jedis.llen("alphabet") == 1
assert jedis.lrange("alphabet", 0, -1) == ["Bravo"]

// del deletes the value associated with the key (no matter the type)
jedis.del("alphabet")
// geting the whole range for a key that doesn't exist safely returns an empty list
assert jedis.lrange("alphabet", 0, -1) == []

Lists also have blocking versions of pop. If the list is empty it will wait till something gets pushed onto the list (left and right versions). This allows you to use redis as a system-wide blocking queue and have a bunch of producer jobs feed work to a number of waiting worker jobs.

Sets

Redis sets have a number of powerful atomic set operations such as union, intersection, and difference

// add a bunch of items to the palindromes set
jedis.sadd("palindromes", "radar")
jedis.sadd("palindromes", "noon")
jedis.sadd("palindromes", "kayak")

assert jedis.smembers("palindromes").containsAll("radar", "noon", "kayak")

// check to see if an item is a member of a set
assert jedis.sismember("palindromes", "kayak") == true

// remove a member from the set
jedis.srem("palindromes", "kayak")

assert jedis.sismember("palindromes", "kayak") == false

// set size (cardinality)
assert jedis.scard("palindromes") == 2

jedis.sadd("acronyms", "scuba")
jedis.sadd("acronyms", "radar")

// set union
assert jedis.sunion("palindromes", "acronyms").containsAll("scuba", "noon", "radar")

// set intersection
assert jedis.sinter("palindromes", "acronyms").containsAll("radar")

// set difference
assert jedis.sdiff("palindromes", "acronyms").containsAll("noon")
assert jedis.sdiff("acronyms", "palindromes").containsAll("scuba")

Each of these methods also has a “store” version (SUNIONSTORE, SINTERSTORE, SDIFFSTORE) that allows you to atomically save the results of the set operation to another key as a new set.

Sorted Sets

Sorted Sets combine the functionality of sets and lists by adding a “score” associated with each element of the set that determines it’s position.

// add months to a sorted set in random order
jedis.zadd("months", 1,  "January")
jedis.zadd("months", 12, "December")
jedis.zadd("months", 5,  "May")
jedis.zadd("months", 9,  "September")
jedis.zadd("months", 3,  "March")
jedis.zadd("months", 10, "October")

// the sorted set returns them in sorted order and can be sliced similarly to lists
assert jedis.zrange("months", 0, -1).toArray() == ["January", "March", "May", "September", "October", "December"]
assert jedis.zrange("months", 3, -2).toArray() == ["September", "October"]

Hashes

Hashes are what the grails redis plugin uses to store domain objects.

jedis.hset("book:1", "title", "Dune")
jedis.hset("book:1", "author", "Frank Herbert")
jedis.hset("book:1", "yearPublished", "1965")

assert jedis.hgetAll("book:1") == [author: "Frank Herbert", title: "Dune", yearPublished: "1965"]

There are also commands to pull out just the keys, just the values, subset of values for particular keys or to check the existence of a key.

Publish/Subscribe

Redis also has some powerful Publish/Subscribe commands that allow the subscriber to listen for new messages on a channel. This allows broadcast messages to be sent out by a publisher to any subscribers of a topic. It’s the compliment to the lists’ ability to act as a blocking queue.

Summary

Redis is a utility belt of functionality and I’ve only scratched the surface of what you can do with it in this post.

At my current startup, we’re looking at using Redis for caching and paginating through some very expensive ad-hoc calculations that are unique to each user of the website. Being able to do this quickly is critical to the user experience, and Redis looks like it has what we need to make this work. The sorted set operations also allow us to easily slice through the data in a way that would normally require something like temp tables in a relational database.

While it can be used as the only data store for an application, I think it’s a perfect compliment for an application with a traditional relational database.

Comments