eZ Community » Blogs » Vincent Robic » Import a million objects from a CSV...


Vincent Robic

Import a million objects from a CSV file without reaching memory limit

Friday 30 March 2012 9:02:15 am

  • Currently 5 out of 5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5


When trying to import objects from a large CSV file into eZ Publish, we often face memory problems. Here is a little tip to avoid it.

To import objects from a CSV file without having any memory problems, you need to unset some global variables.

Indeed, when creating an object, eZ Publish saves it in the $GLOBALS array.

Here is a quick and dirty piece of code to achieve this :

$csvFile = fopen( 'path/to/my/file.csv', 'r' );
while ( ( $data = fgetcsv( $csvFile ) ) !== false )
    if ( $i == 0 )
        $keys = $data;
        $objectData = array_change_key_case( array_combine( $keys, $data ) );
        //Create the object
        //Then, get its ContentObjectID
        $contentobject_id = ...
        //Free memory
        unset( $GLOBALS['eZUserObject_'.$contentobject_id );

This is the most important thing to avoid a memory peak usage. Of course, you must adapt it to the object class you created (an eZUser object in the example). If you are not sure, just use print_r() or var_dump() to explore the content of the $GLOBALS array.

You can also unset these variables, even if it's not very efficient :

 unset( $GLOBALS['eZContentObjectContentObjectCache'] );
 unset( $GLOBALS['eZContentObjectDataMapCache'] );
 unset( $GLOBALS['eZContentObjectVersionCache'] );
 unset( $GLOBALS['eZContentClassAttributeCache'] );
 unset( $GLOBALS['eZContentClassObjectCache']);
 unset( $GLOBALS['eZTranslationCacheTable'] );

These lines are equivalent to eZContentObject::clearCache().

You might create a function to unset all these variables (including $GLOBALS['eZUserObject_<The ID>']), and print the memory usage.

Using this tip can make you able to create 780 000 users with a constant memory load of 19Mb.

Proudly Developed with from