Comments on: Batch Import Performance with Grails and MySQL

By: gaurav

gaurav — Sat, 03 Mar 2012 16:19:49 +0000

wow man………………… The most difficult thing explained in most simple way

By: Better Grails Batch Import Performance with Redis and Jesque - Ted Naleid

Better Grails Batch Import Performance with Redis and Jesque - Ted Naleid — Fri, 14 Oct 2011 04:31:59 +0000

[...] A couple of years ago, I put up a well-received blog post on tuning Batch Import Performance with Grails an MySQL. [...]

By: Bulk data loading into MySQL | Vijay Kandy

Bulk data loading into MySQL | Vijay Kandy — Sat, 24 Sep 2011 05:48:00 +0000

[...] information on improving batch import performance, read Ted Naleid’s post on that topic. September 23rd, 2011 | | 2 views | Category: Groovy | Leave a [...]

By: Jordan Howe

Jordan Howe — Tue, 31 May 2011 17:53:34 +0000

Thanks for this post! I ran across it quite a while ago, and filed it away for future use. I just remembered it and used it to reduce a batch job I occasionally have to run from 2+hours to < 4 minutes. This makes life easier.

By: Leo Sambayan

Leo Sambayan — Wed, 16 Mar 2011 02:44:47 +0000

Thanks for this article. It really helped!

By: Ravi

Ravi — Tue, 15 Mar 2011 17:46:28 +0000

Thanks! Very comprehensive …
Small correction: if ((index+1) % 100 == 0) cleanUpGorm()….to avoid unnecessary commit at 1st record

By: Enrique Medina Montenegro

Enrique Medina Montenegro — Thu, 09 Dec 2010 17:03:42 +0000

I wonder how GC gets impacted by massive creating of objects in Eden segment. Has anyone run into long GC process that blocks application for some seconds?

If so, can you please share GC configuration? Parallel? Old parallel? New concurrent mark sweep? Size for Eden and Mature segments?

Thanks.

By: berlinguyinca

berlinguyinca — Sat, 08 May 2010 01:18:05 +0000

thanks,

still debugging it. It’s getting better, but not perfect yet. One days I will have it perfect.

By: Jaromir D.B. Nemec

Jaromir D.B. Nemec — Wed, 05 May 2010 20:58:20 +0000

Loading mass data in the database using Hibernate is something like transporting people on the highway using motorcycles (one record a time). Of course it is possible to scale between a low class bike and a Fat Boy, but this doesn’t mean this is the right way. The argument about validation of te data with the domain class is valid, but this is neither argument for Grails nor for Hibernate. It should be possible to reuse the validation closure from an external loading process as well.
But anyway this is a great tuning result that many will find useful.

By: tednaleid

tednaleid — Thu, 18 Mar 2010 00:13:14 +0000

@berlinguyinca I could be wrong, but I think you need to take a look at the queries that are getting generated by your code. See how many times for each insert each query is getting called. There might be situations where you could use an executeQuery rather than pulling back and hydrating entire database objects.

You’ll want to beware of the n + 1 problem too, and might be better off doing eager loads of objects rather than lazy loads. It’s hard to get into much more detail without actually seeing code.

If you’re doing hundreds of queries for every insert (even if they’re all really quick to start), that’s going to bog down over time. Looking at (and minimizing) the number of queries might be your best bet.

I’d also look at connecting jconsole to see if memory is going up and then not being released too. That could point to a memory leak on the grails side.

Good luck.