Sunday 26 September 2010 5:05:08 pm
I've created a custom class to hold ISO country names and codes and I'm trying to import an xml file with the data. I've tried a lot of different things to debug but I keep getting an error:
An error occurred during "countryimporthandler" import process : invalid "countryname" field for current SQLIContent.
Country name is just a standard ez text line and I've checked and rechecked to ensure that I haven't misspelled any identifier names. I even tried using the article class instead and putting country name in the title but that didn't work either. Also from the print line I can see that the xml is being parsed correctly.
I'd be really grateful for any help debugging. When this input handler is working I think it might be a good example because it imports the ISO country code list which is handy if you don't want to use the country datatype and would rather use objectrelation.
The XML File (sample)
The INI Override
Name=Country XML Import Handler
The Import Handler
* File containing demo import handler SQLIRSSImportHandler
* @copyright Copyright (C) 2010 - SQLi Agency. All rights reserved
* @licence http://www.gnu.org/licenses/gpl-2.0.txt GNU GPLv2
* @author Jerome Vieilledent
* @version 1.1.0
* @package sqliimport
* @subpackage sourcehandlers
class SQLICountryImportHandler extends SQLIImportAbstractHandler implements ISQLIImportHandler
protected $rowIndex = 0;
public function __construct( SQLIImportHandlerOptions $options = null )
parent::__construct( $options );
$this->remoteIDPrefix = $this->getHandlerIdentifier().'-';
$this->currentRemoteIDPrefix = $this->remoteIDPrefix;
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::initialize()
public function initialize()
$xmlFile = $this->handlerConfArray['xmlFile'];
$xmlOptions = new SQLIXMLOptions( array(
'xml_path' => $xmlFile,
'xml_parser' => 'simplexml'
$xmlParser = new SQLIXMLParser( $xmlOptions );
$this->dataSource = $xmlParser->parse();
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::getProcessLength()
public function getProcessLength()
if( !isset( $this->rowCount ) )
$this->rowCount = count( $this->dataSource->ISO3166Entry );
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::getNextRow()
public function getNextRow()
if( $this->rowIndex < $this->rowCount )
$row = $this->dataSource->ISO3166Entry[$this->rowIndex];
$row = false; // We must return false if we already processed all rows
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::process()
public function process( $row )
// $row is a SimpleXMLElement object
$this->currentGUID = $row->ISO3166Alpha2Code;
$contentOptions = new SQLIContentOptions( array(
'class_identifier' => 'isocountryentry',
'remote_id' => (string)$row->ISO3166Alpha2Code
$content = SQLIContent::create( $contentOptions );
$content->fields->countryname = (string)$row->CountryName;
$content->fields->alpha2code = (string)$row->ISO3166Alpha2Code;
// Now publish content
$content->addLocation( SQLILocation::fromNodeID( $this->handlerConfArray['DefaultParentNodeID'] ) );
$publisher = SQLIContentPublisher::getInstance();
$publisher->publish( $content );
// Free some memory. Internal methods eZContentObject::clearCache() and eZContentObject::resetDataMap() will be called
// @see SQLIContent::__destruct()
unset( $content );
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::cleanup()
public function cleanup()
// Nothing to clean up
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::getHandlerName()
public function getHandlerName()
return 'Country Import Handler';
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::getHandlerIdentifier()
public function getHandlerIdentifier()
* @see extension/sqliimport/classes/sourcehandlers/ISQLIImportHandler::getProgressionNotes()
public function getProgressionNotes()
return 'Currently importing : '.$this->currentGUID;
Sunday 26 September 2010 9:35:06 pm
This error actually comes from an exception thrown when you want to access to an invalid content field (not present in the content object data map).
I can see 2 causes for your problem :
- You misspelled the attribute identifier (<i>countryname</i>, but you tell me that you double-checked this
- You might have de-synchronized objects from their content classes (see issue #14307 - http://issues.ez.no/IssueView.php?Id=14307).
If your are sure that the attribute identifier is correct, try to check for missing object attributes as described in the issue above.
Tuesday 28 September 2010 10:35:07 am
The SQLIImport log just has the same message, that the field is invalid. The error.log file just has a bunch of template override errors (can't find the template file) but I don't think that's related.
I didn't try the script indicated in the post as I had a bit of trouble getting it to run (autoload error). But I did install the ezscriptmonitor extension mentioned as a fix for the same issue. It successfully ran the class synchronization on all the classes I've tried to import too but it didn't help. I'll try to install a clean version of the extension and try the example again. One thing I have discovered is that the non object error seems to go away after clearing all the caches.
Tuesday 28 September 2010 5:35:07 pm
thanks for the extension.
I work with the data_import extension and would like to replace it with sqliimport.
The CSV files in the stubs directory I have found. But, I get it not to work.
The existing RSSimport is an XML example.
Maybe, can you create a new csv example for beginners like me.
e.g new sqlifolderimporthandler.php
Monday 04 October 2010 11:05:09 am
I think I found something, but I'm not sure if this is also affects you...
Can you please try to manually set your <i>activeLanguage</i> ?
$myContent->setActiveLanguage( 'nor-NO' ); // Place here your problematic locale
Of course, repeat this operation for each children if you have ones
Thursday 14 October 2010 12:35:13 am
This may be part of the longer term plan but it would be great if it was possible to have a feature in the admin interface where you could create import mapping settings for classes.
In the settings one would define:
- an existing class (browse to and select)
- the location of an xml or csv file (browse to and select)
- the path (xpath?) for the xml element(s) which represents the container of what will become a class object
- mappings for each class attribute to an xml atomic value (using xpath statement?)
Taking it one step further, it would be great if the import could be able to import an xml hierarchy of elements and preserve the hierarchy of the newly created class objects in the node tree.
How difficult would this be to implement? Is anything like this planned?
Thursday 14 October 2010 9:35:14 am
Your request is interesting, but quite difficult to implement I guess.
If I understand you well, you'd like a feature like in <b>data_import</b>, but more flexible don't you ?
I'm not a fan of defining this kind of settings directly in the admin interface. I'd rather prefer to use a config file, where you could do the mapping.
Moreover, I'm not sure <i>Xpath</i> would be the best choice as you can have many issues when your XML has namespaces for example.
Your idea is really interesting, but I can hardly imagine how to implement it. Could you precise your need a bit please ? Maybe you already have an idea
Friday 15 October 2010 9:05:11 am
I had a use case where the CSV row contains a value that that is used to match to a value of a certain attribute (I did this with a normal fetch). I needed to update a matrix in the content object using all the values in the CSV (separate files with different number of columns).
I did not find a reasonable method to do the following:
- Use the content publisher to modify a specified node by ID
- Loop through all the fields of the row and push the to a string for importing
Should these be possible?
Friday 15 October 2010 10:35:12 am
- Use the content publisher to modify a specified node by ID
Actually you can initiate an <b>SQLIContent</b> object from an <i>eZContentObject</i>, <i>eZContentObjectTreeNode</i>, a NodeID, a ContentObjectID, and of course a RemoteID.
// Assume that $myFetchedNode is an eZContentObjectTreeNode object previously fetched
$mySQLIContent = SQLIContent::fromNode( $myFetchedNode );
Et voilà !
Take a look in the API doc, you'll see other fromXXX methods
Loop through all the fields of the row and push the to a string for importing
<b>SQLCSVRow</b> does implement <b>Iterator</b> interface, so you can smoothly loop against its fields with a <i>foreach</i> loop :
foreach( $myCSVRow as $fieldName => $field )
// Do stuffs with your field
Hope this helps
Friday 29 October 2010 12:35:21 pm
I want to import an HTML code to an attribute Block Xml. The HTML contains several images, like this : <img src="../images/an_image_test.jpg" />
The function SQLIContentUtils::getRichContent() don't allows this, and it delete the img tag.
It will be a great if you can integrate this feature in sqliimport.
1- Browse the html code, and recover img tags,
2- Creating images in BO
3- Add an attribute id to img tag, like this "eZObject_ [contentobject_id of object image]" OR "eZNode_ [node_id of node image]"
Wednesday 17 November 2010 2:35:21 pm
Well just two simple issues :
1 / in the publish method, when you're dealing with the attributes, you check if the fromString method exists in the datatype declaration class. The problem is that all datatype classes extend the ezDataType class witch implements the fromString method who does nothing. So if the specific datatype class doesn't have it's own fromString method, the attribute won't be set !
2/ A documentation for the publish options would be helpful.
that's all, beside does little problems, the extension is really useful.
Friday 19 November 2010 8:35:18 am
I believe I've found a conceptual error in the way the extension works. Namely, one can not set parameters to his source handler in options, and have them override the "defaults" which are configured in the ini settings of a handler (sqliimport.ini.append.php file).
So, for instance, if I invoke this in the command line:
php extension/sqliimport/bin/php/sqlidoimport.php --source-handlers=rssimporthandler --options="rssimporthandler:efaultParentNodeID=65,RSSFeed=http://someotherblog.com/rss"
The SQLIRSSImportHandler will still use the RSSFeed url from your blog, and will still place new objects under the node defined in ini settings (your handler used just as an example, this is also true for any custom developed handler which uses that one's code as it's base/inspiration).
Or, it's more accurate to say that such a thing <b>is</b> possible, but it has to be expected and explicitly handled in the handler; for example like this (code from sqlirssimporthandler.php, function process($row) ):
$parentNodeID = isset( $this->options['DefaultParentNodeID'] ) ? $this->options['DefaultParentNodeID'] : $this->handlerConfArray['DefaultParentNodeID'];
$content->addLocation( SQLILocation::fromNodeID( $parentNodeID ) );
After examining the code, I believe I can also suggest a solution: the handler class should have the $handlerConfArray and the $options arrays "mashed" together (in order: first global defaults, then INI settings, then any passed options from the invocation), so that in actual usage (custom handler code) one only has to use the $this->options['something'] to get at all the handler's options.
From my investigation of the code I believe the actual "mashing" process needs to be carried out in the SQLIImportFactory class. But PHP is really not my language, don't know it's associative array idioms and don't consider myself qualified to suggest the actual code for this.
Thanks for your time, best regards
Thursday 25 November 2010 10:35:31 pm
Found that issue and I don't know If you've fixed it yet.
well here it is in sqlilocation
public static function fromNodeID( $nodeID )
$node = eZContentObjectTreeNode::fetch( $nodeID );
if ( !$node instanceof eZContentObjectTreeNode )
throw new SQLILocationException( "Unable to find eZContentObjectTreeNode with NodeID #$this->nodeID" );
$location = self::fromNode( $node );
Monday 29 November 2010 5:05:42 pm
Actually, this kind of options are not meant to override INI settings. They are made to help you add on-the-fly options for your handler. They are passed to the import handler constructor.
So if you want the RSS handler to support them, you should extend it and add these options support to it
Thursday 02 December 2010 11:35:22 am
Hi Jérôme, thanks for your reply.
Sorry, but I don't quite understand what you mean. What would be a valid "on-the-fly option" for an import handler? Can you give an example of such a thing for your RSS handler?
I ask because to me it seems quite logical and even required by the usability imperative to have the basic handler options (such as the DefaultParentNodeID for your handler) be settable at the handler invocation (either in command line interface or in the admin siteaccess). However, in the extension's current state a user needs to change these options in the INI settings for his import job - and potentially do so for every import job he runs).
That's what I meant when I claimed to have found a "conceptual error" in the way the extension functions now (and BTW that carried absolutely no disrespect for your work, I hope that was understood).
Yes you are right, the handler can be made to support the functionality I'm asking for. But that introduces a need for in-code checks at every use of any option, like I wrote in the original post (pseudocode here):
$my_x_setting = isset( options[the_x] ? options[the_x] : handlerConfArray[the_x]);
This I think is very clunky and error-prone; it would be much simpler and more usable to have a single "options" or "settings" array which includes every config item for the handler, from both the INI and user's invocation (and with the invocation-time settings overriding the INI ones, as would be expected).
Thanks for your time, best regards
Thursday 02 December 2010 4:05:22 pm
I think there is an issue with the isNew() method of SQLIContent class when objects are in trash.
Indeed, the is New() method check if version is 1 and status is draft but when there is an object in the trash with version 1 the status is archived, so isNew() return false.
Thus, when calling SQLIContent::addLocation() with this object the SQLIContentpublisher is called but there is no node mapped with the object (as the object is in trash) and an exception is thrown "Cannot directly add a location to a not-yet-published content.".
Maybe status archived should be considered when defining an object as new
Wednesday 12 January 2011 7:05:20 pm
Has anyone created source and import handlers to import email?
Looking at the ez components code for retrieving mail using IMAP, it doesn't look like it would be too difficult. Unfortunately I don't know php very well at all so I'm not sure exactly how to go about it, but I would be keen to collaborate with someone on it.
Thursday 26 May 2011 12:05:35 pm
It depends. If it is for internal errors (i.e. parsing exceptions, caught by the import framework and not by your handler), this is not possible.
However you can of course write your own logs in your handler via <b>eZLog</b> :
eZLog::write( 'My error message', 'error_2011_05_26.log', 'var/myproject/logs/myimport' );
Tuesday 07 June 2011 9:35:40 am
We have two different handles that should run, but handler number 2 in the defined sequence in the admin interface should not run in paralell to the number 1. Is this the default behaviour with sqliimport?
basically job1 updates lots of fields and job2 depends on the first job to be finished before it starts.
Thursday 09 June 2011 2:35:42 pm
Based on your experience how do you configure your sqliimport_run and sqliimport_cleanup scripts to run?
We need to execute one script (scheduled) hourly and this will not be run unless our crontab has been configrued properly i know. Would you recomend that we set the cronjob to run hourly for this to fully work or should it be run somewhat later than the sceduled job, for instance 5 minutes past the hour? If the latter is the case will this make sure that all "pased" scheduled items will be run as well?
hope you understand what i am asking you.
Friday 19 August 2011 10:05:24 am
I would like to import images from another server. I have an URL to do this but it doesn't work. For the moment, I copy the images on my server and I give the local path to my script.
$contentOptionsImage = new SQLIContentOptions( array(
'class_identifier' => 'image',
'remote_id' => md5($pathtoimage)
$contentImage = SQLIContent::create( $contentOptionsImage );
$contentImage->addLocation( $folderLocation );
$contentImage->fields->name = $row->title;
$contentImage->fields->image = "/data/http/my_site/extension/sqliimport/".$pathtoimage;
$publisher = SQLIContentPublisher::getInstance();
$publisher->publish( $contentImage );
unset( $contentImage );
Is it possible to give an URL like this :
$contentImage->fields->image = "http://www.anothersite.com/".$pathtoimage;
Friday 26 August 2011 11:35:27 am
That's a great extension, nice job!
I'm looking into the extension because it's so much deeper than my own import extension but if I understand it correctly, the object class must be predefined before importing datas, because a specific handler has to be written for a specific import. Concerning the number of attributes, it's quite fine but for instance if you have several ezselection attributes, the options have to exist in the class prior to the import, could it be possible to modify the class, creating/modifying options on the fly while importing datas?
I know it can be coded in the handler but as far as there are a lot of interesting sqliimport classes, is it already planned somewhere (and i missed it), the general purpose being to allow imports to modify content class on the fly in addition to create/update contents?
I think dealing with options and its multilingual versions can be tricky too..
Wednesday 01 February 2012 7:35:43 pm
I am having a problem with a wedged import which i had to quit with "kill".
Now i can't import anything i have this message : "An error has occurred : Another import is already running. Aborting..."
What is the good way to handle this ?
How can i unlock the importRun process and cleanup the mess ?
Thursday 02 February 2012 12:35:39 am
First, you don't need to kill an import script. You can interrupt à running script from the "Import management" in the admin interface.
In order to unlock your import run script, you need to remove entries from <b>ezsite_data</b> table in your database that relates to sqliimport status.
Thursday 02 February 2012 5:35:40 pm
I import some data in an ezmatrix datatype. But if the ezmatrix is empty in the new import, the old value stay... I tried
$content->fields->dates_ouverture = " ";
but it creates an empty line in my ezmatrix field (and in my website it display some empty data...)
Do you know how to owerite the value of a ezmatrix field ???
Friday 03 September 2010 10:35:03 pm
Your extension is really great, however i have experienced some issues when i do the following:
Could you please verify that the following works:
create a new node $mynode in a non-default language (based on site.ini)
IN the same run please add some new subnodes in the same language as $mynode
Are you able to do this sucessfully?
$mynode should then have two children.
Can you verify that this works?
I get some errors saying that "attributes()" function is not correctly instantiated because the variable is not an object.
This does work if it is trying to import in the default language.
Saturday 04 September 2010 9:05:04 am
Please let me know once you have this fixed.
I will then fetch it from you trunk and try to merge it into my project.
Also would you bee so keen as to briefly indicate where you made the changes?
One more questions:
Is there an easy way to set the date for the content when i import it?
It currently only uses the day of the import, not the date which i would like to be able to set from the datasource xml..
Have a good day in France
Saturday 04 September 2010 9:35:04 am
Sure no problem .
I think I found the problem : in <b>SQLIContentFieldsetHolder</b> on line 162, I make use of <i>$object->availableLanguages()</i> and the problem with this method is that it returns langs in your SiteLanguageList (site.ini).
I'm currently trying <i>$object->allLanguages()</i> instead, but there is no in-memory cache for that, so it might make the content creation less efficient...
About your other question, here's a tip : With <b>SQLIContent</b>, you can use all eZContentObject <i>attributes</i> as virtual properties (this is also the case for <b>SQLILocation</b> and eZContentObjectTreeNode) :
// $myContent is an SQLIContent object
$myContent->published = $myCreationDateTimestamp;
$myContent->modified = $myModificationDateTimestamp;
Saturday 04 September 2010 10:05:03 pm
Thanks for really helping out and making this extension better. It really is a pleasure to work with.
I did download the lateste php class version of the contentholder as well as the publisher.
This did not resolv my problem.
I get the following issue when i try to import the subnodes
An error occurred during "myimporthandler" import process : Unable to find eZContentObjectTreeNode with NodeID # 2393
It could be that i am doing something wrong, but i would really appreciate if you could include your process example where you add a node with its subnodes in a non-default language.
I have to admit that i have not donwloaded version 1.01 but instead manually updated the classes you indicate in your reply to my post here
Sunday 05 September 2010 1:35:04 am
I'm glad you enjoy the extension .
Here's the snippet I used to track the issue (my default language was fre-FR) :
$cli->notice( 'Creating parent folder' );
$options = new SQLIContentOptions( array(
'class_identifier' => 'folder',
'remote_id' => 'folder_remote_id',
'language' => 'eng-GB' // Different language than default one (fre-FR)
$folder = SQLIContent::create( $options );
$folder->fields->name = 'English subject';
$folder->addLocation( SQLILocation::fromNodeID( 43 ) );
$publisher = SQLIContentPublisher::getInstance();
$publisher->publish( $folder );
$cli->notice( 'Creating children' );
$commentOptions = new SQLIContentOptions( array(
'class_identifier' => 'comment',
'language' => 'eng-GB'
$folderLocation = $folder->defaultLocation;
$comment1 = SQLIContent::create( $commentOptions );
$comment1->fields->subject = 'Test 1';
$comment1->fields->author = 'Author 1';
$comment1->fields->message = 'Test message 1';
$comment1->addLocation( $folderLocation );
$publisher->publish( $comment1 );
$comment2 = SQLIContent::create( $commentOptions );
$comment2->fields->subject = 'Test 2';
$comment2->fields->author = 'Author 2';
$comment2->fields->message = 'Test message 2';
$comment2->addLocation( $folderLocation );
$publisher->publish( $comment2 );
The error message you mentionned comes from an SQLILocationException thrown from SQLILocation class when you're trying to create a location with an invalid node ID (node cannot be fetched).
Can you please show your snippet ? Can you also give a stack trace (with XDebug for example) ?
By the way, I commited 3 classes (<b>SQLIContentFieldsetHolder</b>, <b>SQLIContent</b> and <b>SQLIContentPublisher</b>. Can you please try with the whole v1.0.1 ?
Sunday 05 September 2010 3:05:04 pm
I updated the sqlcontent class as well as following your example.
However as long as my "non-default" language is not in the default siteaccess i get the same problems.
By adding the language it seems to work fine.
Have you verified that your test works when not having english present in the siteLanguageList in site.ini.append.php?
Just to let you know.. as this is in my case an example which is only for importing new content once, this is not a problem, with regards to scheduling on a live site, this will however be something that should be resolved.. just to make the product even better of course..
BTW adding new dates by following your example works nicely
Thanks for making this a better extension once again.
Monday 06 September 2010 12:35:05 am
I tried with languages that are not present in my SiteLanguageList (I retried with nor-NO just in case). But of course, you will need to add it to your system (Setup => Languages).
Can you please show me an extract of your code, so I can see what may be wrong ?
You can also try this snippet : http://pastebin.com/bz6Vd7wm . Save this snippet in the <i>stubs/</i> directory of the extension and run it with a non-default language (you might alter the snippet for that of course ).
My guess is that with all your past fatal errors, you introduced corrupted content objects in the database. Do you use remoteID ? Because if this is the case, this might be the origin of your problem as the content API would fetch the corrupted objects in the DB with the remoteID instead of creating new ones.
Wednesday 08 September 2010 12:05:04 pm
Hi Jérome !
I have recently been importing data in a location with a section different than the standard one and I just saw that objects imported with sqliimport are always mapped with standard section even if in a location with a different section.
I think that in sqlicontent::create() you should add $sectionID = $parentObject->attribute( 'section_id' ); when there is no section_id defined in sqlicontentoption
What do you think ?
Friday 10 September 2010 1:35:04 pm
I'm currently investigating sqliimport to see if I can use it to replace a good old synchronisation script.
This script synchronize around 50 000 objects in eZ Publish and fetch the objects by calling a webservice several times and each time with a new offset (so that I don't fetch to 50 000 objects in one time).
The problem with your source handler is that you need to fetch all the objects in the initialize() method and process them one by one using getNextRow().
Is there is solution to avoid fetch all the objects in the initialize() method ?
Thanks for your advises
Friday 10 September 2010 4:05:03 pm
Actually you can do whatever you want !
The main thing is to return the right row in <i>getNextRow()</i>.
What you could do is to make your WS calls in <b>getNextRow()</b>. I guess that you need to fetch your objects by packets don't you ? Well in that case, you'll just need to handle an internal counter to decrement for example. When it comes to 0, you recall your WS with an updated offset, and so on.
When you don't have any more objects returned by your WS, just return false as expected.
The only remaining point is the row count. Maybe you can have that count with another WS call ?
Hope this helps