eZ Community » Forums » Developer » Multi-threading of Cron Jobs
expandshrink

Multi-threading of Cron Jobs

Multi-threading of Cron Jobs

Tuesday 13 September 2011 6:28:36 am - 4 replies

Hi,

We are using ezPublish 4.2.

We have a cron job script; which is configured to run every 15 minutes.

The script pulls files from FTP site based on database select query.

The current script reads database table in while loop to find which file has to pulled, this is not multithreaded.

In that case we are finding two issues:

1. Total elapsed time increase

2. Smaller files has to wait a long until long one get over

We would like, is there any way in ezPublish to make cron job multithreaded?

 

Few code snippet;

cronjob.ini

 

[CronjobPart-frequent]

Scripts[]=processscript.php

 

ezPublish.cron

 

0,15,30,45 * * * * cd $EZPUBLISHROOT && $PHP runcronjobs.php frequent -q 2>&1

 

processscript.php

 

$Result=mysql_query("select count(1) from ezpublishfiles where status='IN PROGRESS'"blunk.gif Emoticon;

 

while($result1 = mysql_fetch_array($Result, MYSQL_BOTH))

{

 

  //do some processing and call process file to download file

  processfile($filename);

 }

Thanks

Dhiraj Jain

Tuesday 13 September 2011 9:12:59 am

PHP has no built-in support for multithreading (eZPublish doesn't either).  But there are options - google "PHP multi-threading" - for extensive discussions on the topic.

In a case like this, the easiest is to use exec to fork a child:

 while file
exec( '/path/to/php -f '/path/to/process.php '.$file');
}

process.php would have the code that does the ftp.

The upside is that this will download all files at the same time as a child process.

The downside is, of course, the same.

Modified on Tuesday 13 September 2011 9:14:44 am by Steven E Bailey

Tuesday 13 September 2011 9:49:10 am

PHP has no built-in support for multithreading (eZPublish doesn't either).  But there are options - google "PHP multi-threading" - for extensive discussions on the topic.

In a case like this, the easiest is to use exec to fork a child:

 while file
exec( '/path/to/php -f '/path/to/process.php '.$file');
}

process.php would have the code that does the ftp.

The upside is that this will download all files at the same time as a child process.

The downside is, of course, the same.

Thanks Steven,

I will try the fork approach and will let you know the results.

Regards,

Dhiraj Jain

Tuesday 13 September 2011 10:17:30 am

You can look at the code in the php script updateserachindexsolr in the ezfind extension for a "multithreaded" execution of a php script - in fact it's multiprocess btw. Of course it will execute single-process when run in windows.

Thursday 15 September 2011 3:29:28 pm

Hi All,

Thank you very much for your help.

I have managed to achieve this functionality by adding a wrapper for executing same cron job in parallel/multi-threading mode.

Thanks

Dhiraj Jain

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from