Tuesday 10 April 2012 1:31:48 pm - 9 replies
Hi all,
Short background.
I have a project that needs to load listing data (Job, car, classifieds etc) into a database of some description and then have it search with Apache SOLR because their faceted support is excellent.
I am not a coder but have built hundreds of sites using Joomla and Wordpress.
I have SOLR installed on the server.
I came across EZ P CE and thought I might test it out. Im hoping for some advice from the community.
I thought of using D7 but loading content into the site is proving to be impossible. Each new content field becomes its own table (2 actually) which means a CSV file containing 27 000 records with up to 50 fields each, takes over 5 hrs on a very powerful server. Clearly not acceptable for a site wanting to store 500 000 documents and possibly update/add 50 000 a day.
Joomla simply doesnt have the ability to do this either, you can use Solr to handle simple documents but not a listing type of document where you have specific information like price, type, model, colour etc etc.
So my question is can EZ P CE handle this?
I notice it has EZ find ready to go out of the box which is awesome.
Can you construct different content types (Eg a job listing will have Title, salary, description, contact info, industry etc etc)?
How hard is it to import large CSV files - eg 50000 row files each day.
I see no pricing information for the commercial editions, would love to have a round figure in my head if I need to go that route, so if anyone has a heads up on pricing I appreciate it. Coming from a Joomla background, I like community editions better.
Thanks for any advice.
Tuesday 10 April 2012 2:11:18 pm
I have made one script before, the only problem is that sometimes it requires some tweaks because of memory usage, so maybe you'll need to create a shell script to call this script several times specifying offset and limit, and of course you'll need to adapt this to your needs, check http://share.ez.no/blogs/thiago-campos-viana/tip-google-login-sample-code before, in my script it avoids duplicating content by checking the username, but you can adapt to check for another field, the content object created is a content user with some custom attributes. Your database will remain with the same amount of tables if you create a custom content class, I'm sorry I'm kind of busy, so I can't give any further advice, maybe somebody can help you a bit more.
<?php class Cliente{ public function __construct($data) { $this->Codigo=$data[0]; $this->Situacao=$data[1]; $this->CodigoPai=$data[2]; $this->RazaoSocialNF=$data[3]; $this->RazaoSocialLD=$data[4]; $this->NomeFantasia=$data[5]; $this->Setor=$data[6]; $this->CNPJ=$data[7]; $this->InscricaoEstadual=$data[8]; $this->KmNF=$data[9]; $this->KmLD=$data[10]; $this->EnderecoNF=$data[11]; $this->BairroNF=$data[12]; $this->CepNF=$data[13]; $this->CaixaPostalNF=$data[14]; $this->CidadeNF=$data[15]; $this->UfNF=$data[16]; $this->ResponsavelNF=$data[17]; $this->SetorRespNF=$data[18]; $this->FoneNF=$data[19]; $this->FaxNF=$data[20]; $this->EmailNF=$data[21]; $this->EnderecoComercial=$data[22]; $this->BairroComercial=$data[23]; $this->CepComercial=$data[24]; $this->CidadeComercial=$data[25]; $this->UFComercial=$data[26]; $this->ResponsavelComercial=$data[27]; $this->SetorRespComercial=$data[28]; $this->TelefoneComercial=$data[29]; $this->FaxComercial=$data[30]; $this->EmailComercial=$data[31]; $this->EnderecoLaudo=$data[32]; $this->BairroLaudo=$data[33]; $this->CepLaudo=$data[34]; $this->CidadeLaudo=$data[35]; $this->UFLaudo=$data[36]; $this->ResponsavelLaudo=$data[37]; $this->SetorRespLaudo=$data[38]; $this->FoneLaudo=$data[39]; $this->FaxLaudo=$data[40]; $this->EmailLaudo=$data[41]; } public function getEmail(){ if(strlen($this->EmailLaudo)>3) return $this->EmailLaudo; else if(strlen($this->EmailNF)>3) return $this->EmailNF; else if(strlen($this->EmailComercial)>3) return $this->EmailComercial; else { return null; } } } function genRandomString() { $length = 10; $characters = '0123456789abcdefghijklmnopqrstuvwxyz!@#$'; $string = ''; for ($p = 0; $p < $length; $p++) { $string .= $characters[mt_rand(0, strlen($characters))]; } return $string; } function deleteAll(){ $objects =& eZContentObjectTreeNode::subTreeByNodeID( array( 'Limit' => 500 ), 3179 ); foreach ( $objects as $node ) { $object = $node->object(); //echo 'dasdas '; $object->purge(); } exit; } set_time_limit(0); require 'autoload.php'; $cli = eZCLI::instance(); $script = eZScript::instance(array('description' => ( "eZPersistentObject tutorial.\n\n"), 'use-modules' => true, 'use-extensions' => true)); $script->startup(); $script->initialize(); $options = $script->getOptions( "[offset:][limit:][delete:]"); $user = eZUser::fetchByName('admin'); $userID = $user->attribute( 'contentobject_id' ); eZUser::setCurrentlyLoggedInUser( $user, $userID ); if($options['delete']==1) deleteAll(); $file = fopen("extension/csvimport/clientes.csv","r"); fgets($file); $offset=$options['offset']; $limit=$options['limit']; $current=-1; while(! feof($file)) { $c=new Cliente(explode(';',str_replace('"','',fgets($file)))); $current++; if($current < $offset || $current >= $offset + $limit){ continue; } $email=$c->getEmail(); //$user = eZUser::fetchByEmail($email); //$userID=($user instanceof eZUser)?$user->attribute('contentobject_id'):false; $objects =& eZUser::fetchByName( $c->Codigo ); if($email!=null && !($objects instanceof eZUser)){ // nova importação //if($email){ // atualizar echo "$email\n"; //$user = new ezpObject('user', 3179, 14, 2,false,$userID); $user = new ezpObject('user', 3179, 14, 2); $first_name = strlen($c->NomeFantasia)>3?$c->NomeFantasia:(strlen($c->RazaoSocialNF)>3?$c->RazaoSocialNF:$c->RazaoSocialLD); $last_name = strlen($c->RazaoSocialNF)>3?$c->RazaoSocialNF:$c->RazaoSocialLD; $username = $c->Codigo; $password = genRandomString(); $user->__set("first_name", $first_name); $user->__set("last_name", $last_name); //if(!$userID) $user->__set("user_account", "$username|$email|" . md5("$username\n$password") . "|2"); $user->__set("code", $c->Codigo); $user->__set("situacao", $c->Situacao); $user->__set("codigopai", $c->CodigoPai); //$user->__set("razaosocialnf", $c->RazaoSocialNF); $user->__set("razaosocialld", $c->RazaoSocialLD); //$user->__set("nomefantasia", $c->NomeFantasia); $user->__set("setor", $c->Setor); $user->__set("cnpj", $c->CNPJ); $user->__set("inscricaoestadual", $c->InscricaoEstadual); $user->__set("endereconf", $c->EnderecoNF); $user->__set("bairronf", $c->BairroNF); $user->__set("cepnf", $c->CepNF); $user->__set("caixapostalnf", $c->CaixaPostalNF); $user->__set("cidadenf", $c->CidadeNF); $user->__set("ufnf", $c->UfNF); $user->__set("responsavelnf", $c->ResponsavelNF); $user->__set("fonenf", $c->FoneNF); $user->__set("faxnf", $c->FaxNF); $user->__set("emailnf", $c->EmailNF); $user->publish(); } } fclose($file); $script->shutdown(); ?>
Modified on Tuesday 10 April 2012 2:29:22 pm by Thiago Campos Viana
Tuesday 10 April 2012 3:32:30 pm
Hello Dan, and welcome to the eZ Community.
Different content types is the basis of eZ Publish, through Content classes & attributes, each attribute being of a particular datatype, datatype being fully customizable and extensible to fit your needs: http://doc.ez.no/eZ-Publish/Technical-manual/4.6/Concepts-and-basics/Content-management.
About the enterprise edition, it is honestly much easier if you drop your local representative an email, imho.
Tuesday 10 April 2012 8:54:54 pm
Hello Dan,
As for importing external data, also have a look at SQLIImport; https://github.com/lolautruche/SQLIImport. Developed by one of the eZ Engineering team members. Very powerful, and extendable also.
Regards, Robin
Wednesday 11 April 2012 2:59:03 am
I have run into issues with Ezfind activation.
Trying to run the ezpgenerateautoloads.php file I get errors like this.
Warning: require(Base/src/base.php) [function.require]: failed to open stream: No such file or directory in /home/****/public_html/ezpub/ezpub/bin/php/ezpgenerateautoloads.php on line 33
Fatal error: require() [function.require]: Failed opening required 'Base/src/base.php' (include_path='.:/usr/lib/php:/usr/local/lib/php') in /home/****/public_html/ezpub/ezpub/bin/php/ezpgenerateautoloads.php on line33
When i search the forums and google it tells me i need ezcomponents installed as its a dependency, however the last version I can find of that is 2010 which doesnt sound right to me.
http://ezcomponents.org/download
What am I doing wrong?
Wednesday 11 April 2012 9:26:55 am
Hi Dan,
To help you it will be good to know what version of eZ Publish and eZ Find you've installed so that I can try to replicate and see why isn't working.
If you installed a community edition then eZComponents should be already in your lib/ezc/ folder and they are seen without other steps.
One thing to check, if you created a config.php, is to uncomment the line:
//define( 'EZP_USE_BUNDLED_COMPONENTS', true );
As said in the documentation of that file it's not required but is good for speed.
eZ Components can also be installed system wide using PEAR:
# pear channel-discover components.ez.no
# pear install -a ezc/eZComponents
Anyhow, don't give up, eZ Publish is powerful and flexible and I think could serve your case well.
All the best,
Virgil
Wednesday 11 April 2012 3:37:26 pm
Quote from Dan T :I have run into issues with Ezfind activation.
Trying to run the ezpgenerateautoloads.php file I get errors like this.
Warning: require(Base/src/base.php) [function.require]: failed to open stream: No such file or directory in /home/****/public_html/ezpub/ezpub/bin/php/ezpgenerateautoloads.php on line 33
Fatal error: require() [function.require]: Failed opening required 'Base/src/base.php' (include_path='.:/usr/lib/php:/usr/local/lib/php') in /home/****/public_html/ezpub/ezpub/bin/php/ezpgenerateautoloads.php on line33
When i search the forums and google it tells me i need ezcomponents installed as its a dependency, however the last version I can find of that is 2010 which doesnt sound right to me.
http://ezcomponents.org/download
What am I doing wrong?
Just download community version with ezc, then copy lib/ezc to your installation lib folder.
You must be logged in to post messages in this topic!