eZ Community » Blogs » Vincent Robic » Import a million objects from a CSV...

By

Vincent Robic

Import a million objects from a CSV file without reaching memory limit

Friday 30 March 2012 9:02:15 am

  • Currently 5 out of 5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Hi,

When trying to import objects from a large CSV file into eZ Publish, we often face memory problems. Here is a little tip to avoid it.

To import objects from a CSV file without having any memory problems, you need to unset some global variables.

Indeed, when creating an object, eZ Publish saves it in the $GLOBALS array.

Here is a quick and dirty piece of code to achieve this :

$csvFile = fopen( 'path/to/my/file.csv', 'r' );
$i=0;
 
while ( ( $data = fgetcsv( $csvFile ) ) !== false )
{
    if ( $i == 0 )
    {
        $keys = $data;
    }
    else
    {
        $objectData = array_change_key_case( array_combine( $keys, $data ) );
 
        //Create the object
        ...
 
        //Then, get its ContentObjectID
        $contentobject_id = ...
 
        //Free memory
        unset( $GLOBALS['eZUserObject_'.$contentobject_id );
 
        $i++;
    }
}

This is the most important thing to avoid a memory peak usage. Of course, you must adapt it to the object class you created (an eZUser object in the example). If you are not sure, just use print_r() or var_dump() to explore the content of the $GLOBALS array.

You can also unset these variables, even if it's not very efficient :

 unset( $GLOBALS['eZContentObjectContentObjectCache'] );
 unset( $GLOBALS['eZContentObjectDataMapCache'] );
 unset( $GLOBALS['eZContentObjectVersionCache'] );
 unset( $GLOBALS['eZContentClassAttributeCache'] );
 unset( $GLOBALS['eZContentClassObjectCache']);
 unset( $GLOBALS['eZTranslationCacheTable'] );

These lines are equivalent to eZContentObject::clearCache().

You might create a function to unset all these variables (including $GLOBALS['eZUserObject_<The ID>']), and print the memory usage.

Using this tip can make you able to create 780 000 users with a constant memory load of 19Mb.

Proudly Developed with from