Cheers,
dan

I’d also suggest looking at using a profiler to see where your code is spending most of it’s time. Yourkit is good, but not free. I haven’t had a lot of luck with the free ones out there, but I think yourkit has a trial period.

If that doesn’t get it done, unfortunately, it’s possible that your particular use case is complicated and large enough that grails/GORM isn’t the right tool for the job.

great from you to respond. Well yes we do update/delete/select/insert (it’s a merging tool for several databases)

So all 10k inserts I tend to rebuild the indexes and the auto vacuum daemon is running and taking care of these issues (did not really tune the auto vacuum)

I do think it’s grails and not the db and yes I also use batch statements.

I checked for indexes and monitored the executed queries (p6sql spy + jdbc profiler, which I slightly rewrote to make it work with current version of log4j, I will upload a fork in a week or two) And it looks that I have all the required indexes.

thanks again,

any other ideas? So far there were good, but already taken care off. I really think it’s more of a grails issue. I ordered some SSD’s for the server to speed up index access, but they won’t arrive before mid/end next week. IO Doesn’t not seem to be the bottleneck anyway.

If it’s the database, it could be that you’re missing an index (or need to tune an index that you already have).

I’m less familiar with current postgres installs, but I also know that in some batch programs I created against postgres a few years ago that batching thousands of records together in a transaction was a pretty big help.

If you’re doing any deletion/modification during your import, it could also be that you need to vacuum your database (something that isn’t necessary in mysql).

Only thing, it’s still gets too slow after a couple of 100k inserts, since the performance gets gradually worse over time. It starts at 50ms and is at 200ms after 200k inserts (nothing compared to before flushing the session, but still)

So there is still something amiss with grails + hibernate + DB

any more ideas?

And thank you for this great post ! I will definitely try all of your tips for my batch imports that were very slow and very memory-consuming !

Again than you for sharing this

BTW, I didn’t manage to have p6spy and sqlprofiler working together (no queries are shown in the SqlProfiler console) with Grails 1.2 under Windows Vista). Like j pimmel, I was curious to know if you did manage to have it work and if you change something from what is explained in Mike Hugo ’s article?

Regards,

Fabien

def attributes = rch.getRequestAttributes()
if(attributes) {
attributes.request.setAttribute(key,val)
}
else {
PROPERTY_INSTANCE_MAP.get().put(key,val)
}