Comments on: Batch Import Performance with Grails and MySQL http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/ Groovy, Grails and OS X tips and tricks Sat, 31 Jul 2010 11:43:17 -0500 http://wordpress.org/?v=2.9.2 hourly 1 By: berlinguyinca http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2666 berlinguyinca Sat, 08 May 2010 01:18:05 +0000 http://naleid.com/blog/?p=66#comment-2666 thanks, still debugging it. It's getting better, but not perfect yet. One days I will have it perfect. thanks,

still debugging it. It’s getting better, but not perfect yet. One days I will have it perfect.

]]> By: Jaromir D.B. Nemec http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2663 Jaromir D.B. Nemec Wed, 05 May 2010 20:58:20 +0000 http://naleid.com/blog/?p=66#comment-2663 Loading mass data in the database using Hibernate is something like transporting people on the highway using motorcycles (one record a time). Of course it is possible to scale between a low class bike and a Fat Boy, but this doesn’t mean this is the right way. The argument about validation of te data with the domain class is valid, but this is neither argument for Grails nor for Hibernate. It should be possible to reuse the validation closure from an external loading process as well. But anyway this is a great tuning result that many will find useful. Loading mass data in the database using Hibernate is something like transporting people on the highway using motorcycles (one record a time). Of course it is possible to scale between a low class bike and a Fat Boy, but this doesn’t mean this is the right way. The argument about validation of te data with the domain class is valid, but this is neither argument for Grails nor for Hibernate. It should be possible to reuse the validation closure from an external loading process as well.
But anyway this is a great tuning result that many will find useful.

]]> By: tednaleid http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2598 tednaleid Thu, 18 Mar 2010 00:13:14 +0000 http://naleid.com/blog/?p=66#comment-2598 @berlinguyinca I could be wrong, but I think you need to take a look at the queries that are getting generated by your code. See how many times for each insert each query is getting called. There might be situations where you could use an executeQuery rather than pulling back and hydrating entire database objects. You'll want to beware of the <a href="http://www.pbell.com/index.cfm/2006/9/17/Understanding-the-n1-query-problem" rel="nofollow">n + 1 problem</a> too, and might be better off doing eager loads of objects rather than lazy loads. It's hard to get into much more detail without actually seeing code. If you're doing hundreds of queries for every insert (even if they're all really quick to start), that's going to bog down over time. Looking at (and minimizing) the number of queries might be your best bet. I'd also look at connecting jconsole to see if memory is going up and then not being released too. That could point to a memory leak on the grails side. Good luck. @berlinguyinca I could be wrong, but I think you need to take a look at the queries that are getting generated by your code. See how many times for each insert each query is getting called. There might be situations where you could use an executeQuery rather than pulling back and hydrating entire database objects.

You’ll want to beware of the n + 1 problem too, and might be better off doing eager loads of objects rather than lazy loads. It’s hard to get into much more detail without actually seeing code.

If you’re doing hundreds of queries for every insert (even if they’re all really quick to start), that’s going to bog down over time. Looking at (and minimizing) the number of queries might be your best bet.

I’d also look at connecting jconsole to see if memory is going up and then not being released too. That could point to a memory leak on the grails side.

Good luck.

]]> By: berlinguyinca http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2597 berlinguyinca Wed, 17 Mar 2010 21:52:07 +0000 http://naleid.com/blog/?p=66#comment-2597 it's me again, still working on the problem and managed to drop the curve a bit with the use of a couple of SSD's in our db server, but still not enough to make it feasible. basically grails seems to store some more data somewhere, i guess. Reason that I'm thinking this is that I can import 100k objects with there relations in a fairly quick time of 100-200ms an insert, till I hit a certain point and than the curve jumps up again. this is my curve for about 1.2 Mio inserts. http://dl.dropbox.com/u/5396494/curve.png The problem is defiantly on the grails side or my rather simple algorithm, since there is now load on the DB during this progress or queries. it’s me again,

still working on the problem and managed to drop the curve a bit with the use of a couple of SSD’s in our db server, but still not enough to make it feasible.

basically grails seems to store some more data somewhere, i guess. Reason that I’m thinking this is that I can import 100k objects with there relations in a fairly quick time of 100-200ms an insert, till I hit a certain point and than the curve jumps up again.

this is my curve for about 1.2 Mio inserts.

http://dl.dropbox.com/u/5396494/curve.png

The problem is defiantly on the grails side or my rather simple algorithm, since there is now load on the DB during this progress or queries.

]]> By:   Rendimiento en proceso de importación con Grails y MySQL (inglés) - Observatorio de Grails http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2587   Rendimiento en proceso de importación con Grails y MySQL (inglés) - Observatorio de Grails Fri, 12 Mar 2010 11:08:59 +0000 http://naleid.com/blog/?p=66#comment-2587 [...] información sobre Batch Import Performance with Grails and MySQL (traducido al [...] [...] información sobre Batch Import Performance with Grails and MySQL (traducido al [...]

]]> By: Blog bookmarks 03/10/2010 « My Diigo bookmarks http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2584 Blog bookmarks 03/10/2010 « My Diigo bookmarks Wed, 10 Mar 2010 04:30:45 +0000 http://naleid.com/blog/?p=66#comment-2584 [...] Batch Import Performance with Grails and MySQL – Ted Naleid [...] [...] Batch Import Performance with Grails and MySQL – Ted Naleid [...]

]]> By: Grails MySQL Batch Insert http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2583 Grails MySQL Batch Insert Tue, 09 Mar 2010 18:06:27 +0000 http://naleid.com/blog/?p=66#comment-2583 [...] stumbled upon this post http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/.  It’s an excellent read and gives some insight into some of the inner workings of both [...] [...] stumbled upon this post http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/.  It’s an excellent read and gives some insight into some of the inner workings of both [...]

]]> By: Dan Lynn http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2581 Dan Lynn Tue, 09 Mar 2010 01:29:37 +0000 http://naleid.com/blog/?p=66#comment-2581 Excellent post. I can't believe you had the patience to gather so much data! Cheers, dan Excellent post. I can’t believe you had the patience to gather so much data!

Cheers,
dan

]]> By: tednaleid http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2557 tednaleid Tue, 23 Feb 2010 02:34:27 +0000 http://naleid.com/blog/?p=66#comment-2557 @berlinguyinca If you haven't done it already, I'd fire up jconsole (or jvisualvm, either should be in your path already if you're using java 1.6) and connect to the grails process. Let it run for quite a while and see if it looks like the memory is continuing to go up and up, or if it sawtooths up and then comes back down to the same level again. I'd also suggest looking at using a profiler to see where your code is spending most of it's time. Yourkit is good, but not free. I haven't had a lot of luck with the free ones out there, but I think yourkit has a trial period. If that doesn't get it done, unfortunately, it's possible that your particular use case is complicated and large enough that grails/GORM isn't the right tool for the job. @berlinguyinca If you haven’t done it already, I’d fire up jconsole (or jvisualvm, either should be in your path already if you’re using java 1.6) and connect to the grails process. Let it run for quite a while and see if it looks like the memory is continuing to go up and up, or if it sawtooths up and then comes back down to the same level again.

I’d also suggest looking at using a profiler to see where your code is spending most of it’s time. Yourkit is good, but not free. I haven’t had a lot of luck with the free ones out there, but I think yourkit has a trial period.

If that doesn’t get it done, unfortunately, it’s possible that your particular use case is complicated and large enough that grails/GORM isn’t the right tool for the job.

]]> By: berlinguyinca http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/comment-page-1/#comment-2556 berlinguyinca Mon, 22 Feb 2010 21:45:11 +0000 http://naleid.com/blog/?p=66#comment-2556 hi, great from you to respond. Well yes we do update/delete/select/insert (it's a merging tool for several databases) So all 10k inserts I tend to rebuild the indexes and the auto vacuum daemon is running and taking care of these issues (did not really tune the auto vacuum) I do think it's grails and not the db and yes I also use batch statements. I checked for indexes and monitored the executed queries (p6sql spy + jdbc profiler, which I slightly rewrote to make it work with current version of log4j, I will upload a fork in a week or two) And it looks that I have all the required indexes. thanks again, any other ideas? So far there were good, but already taken care off. I really think it's more of a grails issue. I ordered some SSD's for the server to speed up index access, but they won't arrive before mid/end next week. IO Doesn't not seem to be the bottleneck anyway. hi,

great from you to respond. Well yes we do update/delete/select/insert (it’s a merging tool for several databases)

So all 10k inserts I tend to rebuild the indexes and the auto vacuum daemon is running and taking care of these issues (did not really tune the auto vacuum)

I do think it’s grails and not the db and yes I also use batch statements.

I checked for indexes and monitored the executed queries (p6sql spy + jdbc profiler, which I slightly rewrote to make it work with current version of log4j, I will upload a fork in a week or two) And it looks that I have all the required indexes.

thanks again,

any other ideas? So far there were good, but already taken care off. I really think it’s more of a grails issue. I ordered some SSD’s for the server to speed up index access, but they won’t arrive before mid/end next week. IO Doesn’t not seem to be the bottleneck anyway.

]]>