eZ Community » Forums » Developer » [eZFind] Solr refuses to index
expandshrink

[eZFind] Solr refuses to index

[eZFind] Solr refuses to index

Wednesday 24 December 2014 11:13:11 am - 3 replies

Hi,

I have a problem with Solr.

I have an import script that creates content in eZ.Creating contained in eZ causes indexing.

After some (time / number of indexed object) Solr crach and refuses any indexation has less then re-start.

 

I hack eZFind to add sone debug.

class eZSolrBase

{
    function deleteDocs ( ... )
    {
        //...
        $updateXML = $this->postQuery ( '/update', $postString, 'text/xml' );
        /////////////////////
        if ( empty( $updateXML ) ) {
            SLogger::e( 'empty $updateXML. '.(time()-$start).' sec' );
            return false;
        }
        /////////////////////
        //...
        return self::validateUpdateResult( $updateXML );
    }
}

 

A the begining of my import script i disabled indexing for prevent Solr slow the process.

 

# Disable indexing
$ezfindINI = eZINI::instance('ezfind.ini');
$ezfindINI->setVariable('IndexOptions', 'OptimizeOnCommit', 'disabled');
$ezfindINI->setVariable('IndexOptions', 'DisableDirectCommits', 'true');
$siteINI = eZINI::instance("site.ini"blunk.gif Emoticon;
$siteINI->setVariable('SearchSettings', 'DelayedIndexing', 'enabled');

 

// Lot of create/delete

 

// And suddenly. On a delete : 

empty $updateXML. 40 sec

 

Solr takes too long to answer and my script eats a timeout.

 

Unfortunately, it dont stop there, because those 40 seconds were enough to my database to disconnect

EXCEPTION : eZDBException
CODE      : 0
MESSAGE   : MySQL server has gone away
FILE      : /var/www/ez-common/originals/ezpublish-2012.6/lib/ezdb/classes/ezdbinterface.php
LINE      : 1045

 

As I disable indexing at the beginning of the script. it's the Frequent that has responsibility.

 

But after this accident it always send me : 

/ezfind/classes/ezsolrbase.php 383 empty $updateResult. 40 sec
        Failed indexing object ID #783766, keeping it in the queue.
Fatal error: A database transaction in eZ Publish failed.
The current execution was stopped to prevent further problems.
You should contact the System Administrator (administrateur@notaires.fr) of this site.
The current transaction ID is TRANSID-0c42911b81bf069a1d4e34408b44d4fd and has been logged.
Please include the transaction ID and the name of the current script when contacting the system administrator.

 

And until I restart Solr.

----------

Env : 

  • ezpublish-2012.6
  • Solr 3.1
  • PHP 5.4.4

Modified on Wednesday 24 December 2014 11:52:58 am by Rémy PHP

Thursday 25 December 2014 3:27:05 am

Hello Rémy,

I would try to disable the default eZ Publish cronjobs from running before starting your import.

I hope this helps!

Cheers,
Heath 

Tuesday 30 December 2014 4:26:00 pm

hum... Why not.. 

Both are cronjobs.. 

Can a cronjob be run while an other already running ? 

Modified on Tuesday 30 December 2014 6:37:18 pm by Rémy PHP

Wednesday 31 December 2014 9:11:44 pm

Hello Rémy,

While cronjobs can be run concurrently in your situation your running into a database performance bottleneck.

Data import processes like yours are often very performance and resource intensive operations.

You have helped reduce this impact already but using the 'DelayedIndexing' feature but for you in this situation your still running into a database performance problem when running both cronjobs concurrently.

In short your database server is clearly unable to withstand your resource gobbling data import load -and- index the imported content at the same time.

You could work on this performance problem (within your data import cronjob) to ensure it does take the lion's share of the database available resources but I think the faster option is to simply disable your other crontab cronjobs while importing since this sounds like a one time import.

When your done importing data then manually run your search index update script (this works well since you loose nothing already since your using the 'DelayedIndexing' feature). Then re-enable your crontab cronjobs.

Again this is not a problem with ezfind or ezpublish, it's a database performance / connection problem.

I hope this helps!

Cheers,
Heath

Modified on Wednesday 31 December 2014 9:12:34 pm by // Heath

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from