eZ Community » Forums » Install & configuration » Removing objects takes loooong time,...
expandshrink

Removing objects takes loooong time, can't seem to disable solr indexing for it.

Removing objects takes loooong time, can't seem to disable solr indexing for it.

Tuesday 12 January 2010 6:46:58 pm - 8 replies

I'm using EZPublish 4.2.0 and EZFind 2.1.

We are currently in development, and have some product classes with thousands of objects (one with > 10,000). When we need to remove them for a reimport, it can take hours.

Importing was slow, too, but since I set DelayedIndexing=enabled, that is now very speedy. That change did not affect object removal speed.

I tried setting OptimizeOnCommit=disabled in ezfind.ini, and this had no effect on object removal speed, either. (Also tried clearing all caches, no luck).

Viewing processes, Java is topping out the CPU during object removal, so solr is clearly still being engaged.

Is there a way to disable the re-indexing, or is it slow like this for everyone, and this is something I should just get used to?

Modified on Friday 15 January 2010 3:38:27 pm by Chris Nelson

Saturday 16 January 2010 10:24:39 am

Hi Chris,

Disabling the re-indexing means that after you purged and imported your products, the solr index file is not consistent any more. So this would require a clean re-index, which in turn would most likely run for some time as well. Is disabling the re-indexing really an option?

You might want to keep an eye on the following thread:
http://share.ez.no/forums/extensions/ez-find-update-index-clean-taking-too-much-server-resources

Saturday 16 January 2010 4:28:42 pm

Thanks for the reply!

The re-indexing is very fast, actually, when done after-the-fact. For example, DelayedIndexing is enabled, and that prevents indexing during the import, and I do a re-indexing pass after that which is very speedy (just a few minutes). It's only the removal that is still slow (2-3 seconds per object removed!), and Java resources spike during the object removal, so I'm convinced that Solr is doing work on the backend, even though DelayedIndexing is enabled and OptimizeOnCommit is disabled.

So, delaying re-indexing is not a problem...

Saturday 16 January 2010 8:02:44 pm

Would it be worth a try to completely de-activate eZ Find? See http://ez.no/doc/extensions/ez_find/2_1/installation, same way as you activate the extension. Of course just before you purge and re-import all of your products?

I have no idea on the impact of disabling eZ Find, but if you do a re-indexing anyway, I don't think it's going to harm your setup.

-- Robin

Saturday 16 January 2010 9:47:14 pm

Removing those objects through the API calls a delete and commit with every object to the backend index. This can be indeed be quite slow when there are many objects.

Can you file an issue for this? In the current trunk, there are optimisations for high write traffic sites, I'll add a config option to eZ Find that disables commits on delete and add a small script that you can use after the delete operations are finished to issue a commit.

(warning: before the commit, deleted objects will still show up in the search and trigger fatal errors as the eZP side calls will fail obviously)

@Robin: disabling ezfind alltogether during this delete will indeed speed up the whole operation as well

An alternative would be to patch Solr, so it also accepts a "commit within" parameter like it does for updates/additions ... I'll have a look at that too, the Java code to accomplish this is quite simple at first sight.

hth

Paul

Sunday 17 January 2010 9:01:11 am

@Paul, thanks happy.gif Emoticon

@Chris, you can file an issue here: http://issues.ez.no/HomePage.php? . Tip; when filing the issue, link back to this forum thread.

Thanks, Robin

Monday 18 January 2010 3:18:56 pm

@Paul, thanks happy.gif Emoticon

@Chris, you can file an issue here: http://issues.ez.no/HomePage.php? . Tip; when filing the issue, link back to this forum thread.

Thanks, Robin

Thank you for your responses. I will file the issue today, after I'm out of meetings.

Wednesday 24 February 2010 12:36:09 pm

Dear Paul,

do you have any news about the script or the patch you mentioned above?

I have too the problem described by Chris, using ezfind 2.1 on ez 4.2

Many thanks,

Gabriele

Monday 01 July 2013 5:18:34 pm

I would like to reopen above discussion, as I've got the same problem. Daily import script tries to remove ~8000 nodes, which takes many hours.

Solution implemented for this problem doesn't seems to work, here is link to the ticket with my comment:

https://jira.ez.no/browse/EZP-16372?focusedCommentId=77896&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-77896

If anybody has any tip for that problem, please help.

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from