This site has been archived. To learn more about our current products Ibexa Content, Ibexa Experience, Ibexa Commerce head over to the Ibexa Developer Portal

eZ Community » Forums » Developer » eZ - what is ripping our resources?
expandshrink

eZ - what is ripping our resources?

eZ - what is ripping our resources?

Wednesday 24 March 2004 2:10:17 pm - 30 replies

I LOVE when people leave their work, the evening/night is getting closer..the visitors of our website are almost gone and are going home for the evening, back to their families..and the awful slow eZ site is getting back to "normal"(a usable shape where processing only takes from 2-8 seconds, and it's possible to use the web again).

IT's LOVELY. We ran other sites on the same server earlier, but had to move everything to another server, because of the big resource-need eZ publish is having.

I hereby ask for some tuningtips of eZ publish. Haven't found this doc. anywhere on www.ez.no

The eZ Systems team reccomend using a PHP-accellerator.
Since this is absolutely neccessary to run an eZ publish site with "ok" results, I need tuning tips. We use Turck mmcache, but there must be any good ez docs on this (made by ez, so the eZ team know how to best tune the customers sites). Is this available eZ?

Which settings is the best for an eZ publish site (for a small site, medium site and a large site), how is it possible to administrate this accellerator(empty cache(implement in cron-jobs for clearcache), tune, conflicts-checking etc.).

We're running eZ publish 3.3 and it's making us crazy.. Some of the editors are complaining, and tell me the publishing is taking too much of their time..they're waiting and waiting...and the web stands still when they publish. Not just in the admin-part, but the whole web. Is it better to change CMS-system than develope and develope a system which has given us a slow-site from the very beginning?

We have checked out the database, and one "thing" that Emil Webber also said(in another posting) was, that there are missing A LOT of indexes in the db which make eZ publish REALLY SLOW.

Shall we(the eZ customers make the indexes ourself), or shall this actually be a part of eZ publish Content Management System?

We bought a complete eZ publish solution(also got ez to tune our site after every upgrade they've done), but we still got problems.

The whole web hangs when people publish content. We got a large db, many editors, very heavy trafic, our hardware is very good, and we got 1 GB connection. All the other sites are running wonderful, and this is the first time we've got problems with any php-mysql application ever.

PLEASE eZ, can you soon post any doc. with tuning tips on the PHP accellerators you recommend for running eZ "stable", and can you give us an overview of the databasemodel, and where it's important to make indexes in the database for speeding up the site.

Tnx.

Wednesday 24 March 2004 9:25:18 pm

Howdy,

I would highly suggest Zend (www.Zend.com). Zend Accelerator would give you a significant performance boost (20-400%) by pre-processing your php files. You might also consider the Performance Suite, which would also give you high performance caching.

This is the system that we're offering to our clients with eZ Publish, and it's pretty much the best php accelerator on the market.

Scalability with eZ Publish is definitely a problem that is going to need to be addressed more than it has been if eZ Publish is going to continue making inroads into the more Enterprise CMS space.

Jonathan

Wednesday 24 March 2004 10:01:54 pm

I know..but, we've tried different PHP accellerators..and our latest one is the one from Turck..but the whole ez publish system is just totally "weird".

Why so much queries to get data, and to update data in the database? I think a solution is that eZ publish has to be rewritten regarding the queries and so on.

There are so many CMS-systems on the market. They offer very much the same functionallity as ez publish, so if the ez team want their customers to be satisfied, they really have to make some changes.

Gnite

Modified on Thursday 25 March 2004 9:18:23 am by K259

Thursday 25 March 2004 9:23:02 am

Hi Zinistry

Sounds frustrating to say the least!

Do you have some specifics? When you say "We got a large db, many editors, very heavy trafic, our hardware is very good, and we got 1 GB connection" -

What are the server specs?
How large is the DB?
How many objects are in the database?
How many hits per hour are you getting?
What OS are you using?
Is the database and the webserver on the same system?
Do you use cache-blocks?

There are plenty of questions to ask and areas to look at.

Would be happy to chat off the forums - <brucem@designit.com.au>

Cheers
Bruce

Thursday 25 March 2004 12:26:09 pm

Server
Dell 2650, 2x1.6 GHz
Pentium 4 Xeon cpu's, 2GB RAM, and 200GB diskspace on RAID5 controller.

OS
Redhat 9

Database: 800mb

select count(*) from ezcontentobject = 19049
select count(*) from ezcontentobject_attribute = 332778

Total hits february: 10450185
Total visits february: 255241

80 editors, and the whole system(not only the admin-part) hangs when they are publishing content.

20-25 roles, 20 sections

eZ have set up the the system for us, and done the upgrade to the 3.3 v., and they've used cache-blocks in the templates yes. They've checked our hardware and software, our configuration etc. many times, but I haven't got any answers of what we can do. Have had speed-problems since we started using ez publish, and after a change in the role-system(they told me there was a bug here half a year ago or so, and something was changed), and now the publishing process stops the whole web. It was not like this earlier, but I don't have a clue why this is happening. To heavy queries which locks the queries in mysql, or what? hmm..

I'm checking the indexes in the database now, and will add some to check if it helps in the select-processes. Unfortunately there are almost no documentation on tuning ez, adding indexes, configuring PHP accellerators on www.ez.no

Our experiences are that everything hangs when publishing content, once the publishing-process is finished, then the web works again.

The time it takes to publish content is from 3 seconds, and up to around 30 sec.

I'm interested in getting in touch with companies running 3.x sites with a minimum of a 500 mb database, many editors, many sections, many roles and some traffic. It would be nice to check out your 3.x site and check the speed etc.

Modified on Thursday 25 March 2004 12:33:41 pm by K259

Thursday 25 March 2004 2:21:52 pm

Hi,
I have had a lot problems with ez on our servers as well. Specially when creating content through a php script to import a lot of rows from "flat-files". This process always stopped after creating about 250 rows of data, so we had to split the files up in smaller bits to get it running (so 3000 rows became 12 files). I think there maybe some memoryleak that clogs up the system when publishing a lot of content at the same time, or in a sequence. There is no problem when our small amount of users is browsing and using the website rest of the time. But the problem got me thinking. Pretty strange that its not using indexing also! We also have very good hardware. 1 GB memory or so, double processors etc... running redhat 8.

Have you checked your php version? i know that 4.2.2 (was it?) is not a good one with ez.

cheers,
claÜs

Thursday 25 March 2004 3:37:22 pm

Well..small amount of content is going much faster than if the text which is beeing published is very long..but I've never seen anything like it in other systems.

We have always followed the recommendation from the ez team.

We're dropping to go for apache 2 which we actually would like to do, but this is not recommended with ez publish yet, so we wait.

We're running PHP 4.3.4 with Turck MMCache v2.4.6.

Thursday 25 March 2004 3:56:48 pm

Just want to clearify this:

It is not eZ publish that has a problem with Apache 2.x, this is a PHP problem. From a previous thread:

"The main problem is that Apache 2 is using threads, but PHP is not thread safe. This means that strange things can happen when serving multiple requests. With very low load and one concurrent user this should not be a big problem, but on larger sites with much traffic you can have problems."

Balazs

Thursday 25 March 2004 4:14:24 pm

Yes, I remember Balazs.

But what about the PHP versions? Is eZ publish stable with both v. 4.2.x and 4.3.x?

Does it exist any ez-doc on tuning, how to config PHP-accellerators, creating indexes on the most important places in the db etc.?

Thursday 25 March 2004 4:29:44 pm

Hi again,

There is a problem with 4.2.x - a nasty session bug that might cause very weird behavior. The recommendation/requirement is to use 4.3.2 or later (but not 5.x.x) - this is clearly outlined in the doc pages:

http://ez.no/ez_publish/documenta...ents_for_doing_a_normal_installation

Tuning:
I believe that there exits a page here and there (cache blocks for example), but not a complete manual that outlines all possible tuning tricks. I guess we'll have to write a small chapter about this; in the future. In the meantime: don't hesitate to use the forum.. ask & nag; that's what we have the forum for. happy.gif Emoticon

Indexes:
The missing indexes will be added in upcoming releases. However, these must also be tested extensively; too many indexes might result in slower publishing. Stay tuned.

Balazs

Modified on Thursday 25 March 2004 4:31:25 pm by Balazs Halasy

Friday 26 March 2004 12:43:38 am

> Our experiences are that everything hangs when publishing content, once the
> publishing-process is finished, then the web works again.
>
> The time it takes to publish content is from 3 seconds, and up to around 30 sec.

To me this sounds like the updates are waiting to get a lock on the table. i.e. they have to wait until all the selects have finished.

From http://www.mysql.com/doc/en/Table_locking.html
"To achieve a very high lock speed, MySQL uses table locking (instead of page, row, or column locking) for all storage engines except InnoDB and BDB. For large tables, table locking is much better than row locking for most applications, but there are some pitfalls."and"Table locking enables many threads to read from a table at the same time, but if a thread wants to write to a table, it must first get exclusive access. During the update, all other threads that want to access this particular table must wait until the update is done."

and

"Table locking is also disadvantageous under the following scenario:
* A client issues a SELECT that takes a long time to run.
* Another client then issues an UPDATE on a used table. This client will wait until the SELECT is finished.
* Another client issues another SELECT statement on the same table. As UPDATE has higher priority than SELECT, this SELECT will wait for the UPDATE to finish. It will also wait for the first SELECT to finish! "

Zinistry, thanks for all the information, the one thing you didn't specifically answer is weather the database is on the same server as eZ - I get the feeling that it is.

I'm pretty confident that given the situation you have mentioned that the database will be the source of the problem.

Things you could look at -

+ check and optimize all tables Look at http://www.mysql.com/doc/en/myisamchk_syntax.html
+ Change the storage engine from the default to one that supports row locking such as InnoDB http://www.mysql.com/doc/en/InnoDB_overview.html

"InnoDB provides MySQL with a transaction-safe (ACID compliant) storage engine with commit, rollback, and crash recovery capabilities. InnoDB does locking on row level and also provides an Oracle-style consistent non-locking read in SELECT statements. These features increase multi-user concurrency and performance."

I do not know how this may effect ezPublish but it may be worth a try

+ use PostgreSQL - I take it that it is still supported?

+ Look into load balancing the database - admin off the master - front end off the slave(s)
http://www.mysql.com/doc/en/Repli...oad_balancing_and_useslaveservertrue
http://ez.no/community/forum/install_configuration/enterprise_setup

Plenty to look at but I would focus on the database.

Cheers
Bruce

P.S. I'm also interested in other large sites.
Our largest ez site had 45,543.00 session in Feb 2 editors and a DB of 42Mb ( a shadow of what you are dealing with) Would be interested to know about the ez setup!

Friday 26 March 2004 9:46:16 am

I was reading your thread...

My frist estimate is that the db is the main problem. The are many switches you can turn. Bruce already provided the proper URLs.

We consulted a site for eZ publish

They have about <b>200000 pageviews</b> a day, but their DB should be below 5 MB

The server is a 3 GHz XEON with 1GB RAM
Each request takes about 0.11 sec
The server can deliver about 9-10 requests per sec

We spend about 3 days optimizing the server (Mysql,php, apche to our needs)
<b>The main problem on this server was the apache was using a lot of ram so that the swap got filled. The large swap solwed down mysql queryies.</b>

Modified on Friday 26 March 2004 10:30:14 am by Björn Dieding@xrow.de

Friday 26 March 2004 3:32:27 pm

Hi,

a little story:
This week I made an ez "hack": I've changed the behaviour of the "url-alias". German "umlauts" (and other special characters are now translated in a better way, e.g. ö => oe and instead of using the underline character, I use the "-". (Google like this.)

But the main reason was: I made a small site for a client who publishes content belonging to "homeopathy". The german word of "homeopathy" is "Homöopathie". And the translated url was - in a normal ez installation e.g.
/seminare/hom_opathie

"Homo_pathie" -> The first part of the word is "Homo" which can be translated to "gay" in english. And my client wasn't "amused".

Okay, the update worked fine, all new urls are working. But I've overseen something: The speed of the search engine spiders. 2 hours after the upload of the hack my server was under very heavy pressure: Average server load of 20 up to 40. Oh no!

The reason: 5 spiders (google, yahoo and three others) were crawling all my sites without remorse. The cache wasn't filled (because of the change of the urls I had to delete all my cache ...) So I got 45.000 page views only from spiders at this day (!).

Small change - big effect happy.gif Emoticon

All in all I can say:
The main reasons why ez3 is slow:
-) The template engine is [*censored*]. The ez crew is knowing that, but unfortunately the "template compiler" isn't stable yet.
-) The ez3 caching system is [*censored*] -> now I am using an own "add-on" which stores the cache-input in the db. (Otherwise I've around 100.000 cache-files which are deleted if someone publishes content ...) -> see my other postings.
But the cache-db-addon is - of course - only helping if the cache is filled with data. (In my case, the cache was empty). And cache is only helping if many users are viewing the same page again and again (the search spiders were crawling only "new pages"blunk.gif Emoticon.

Tuning the server (apache / mysql) should be the last step. Of course it's necerssary and important, but it shouldn't be the first step.

-> Gnite: If you want to exchange experiences, my email is emil.webber@gmx.at

Kind regards,
Emil.

Monday 29 March 2004 6:41:28 pm

Tnx for alle the replies on this topic.

I agree with you Emil. Tuning the server (apache / mysql) should be the last step, and the CMS-system itself should be ok to run like all other systems without any big changes in the config of PHP or Mysql(what about the ppl which have very strict access to their hosting-server(through their ISP) and do not have the possibillity to make all these changes, checks and tuning??), and if we want the extreme results of performance, we have to do some tuning..of course. ...but is it eZ publish, or is is PHP and Mysql which gives us all these performance-problems`? I have never earlier experienced anything like this with other CMS-systems.

I'm going to add some indexes tonight, check out database-settings and then see...

The eZ team have checked out our configuration many times earlier, both PHP, Apache and Mysql, settings, versions etc. etc., and they have not found anything incorrect, but the system has been unstable and very slow from the very beginning". Hopefully the new changes in 3.4 will make things better?

Don't think the indexes will help btw., since our website hangs a lot of seconds every time content is beeing published. BUT I'm going to analyse the tables, check out our settings ONCE again...... :/

And a question for the eZ team: How much does it cost to get an eZ site tuned for the best performance with PHP and Mysql? (I'll bet more ppl wanna know the costs also). OR, if you don't wanna give the price here, how many hours do you need to do this job?

Maybe one of the eZ partners can do this job?
I'm thinking about giving "you" another chance here if the costs is not too high. Since we already have been using too much resources on this site since we started using this system. We need this "thing" stable before we're going to have another 25 editors on course with content-publishing.
If not the system is stable, I guess the whole system goes down like the other times we had these courses..

Modified on Monday 29 March 2004 6:47:57 pm by K259

Monday 29 March 2004 7:12:42 pm

Zinistry,

In our experience it is unlikely to be the database that causes the bottleneck. We normally find the database uses about 3% of the CPU time.
It seems to me that part of the problem is that when your editors publish the system needs to regenerate the cache every time. This means that the accelerators will be useless as they will be constantly recacheing cahce pages.

You could try turning off the cache within eZ and see if this helps with things. That way you will not be constantly deleting and recreating cache files. This would only be a stop gap solution, hopefully 3.4 will solve some of your issues as they are reducing the memory payload of eZ.

oh, just a thought, when was the last time this box was patched? I know that in December Redhat patched their 2.4 kernel with some 2.6 kernel features. This might speed things up. I know it did for us.

Hope this helps

--tony

Tuesday 30 March 2004 1:45:44 am

Hi Zinistry,

To rule out the database you could run a "show processlist" from mysql while the pause is occurring. http://www.mysql.com/doc/en/SHOW_PROCESSLIST.html This should give a snapshot as to what the database is doing during these times. Also a "show status" may shed some light on any issues.

RedHat 9 should be running "sar" by default. sar collects, saves and reports system activity."sar -A" will give details of system stats and could be used to rule out any system related issues (swapping, cpu load, memory usage...etc)

There are many things that could be causing this issue. Default installs of applications like mySQL are NOT optimal for every situation. For example mysql has a default wait_timeout value of 28800 secs (8 hours) This value is the number of seconds the server waits for activity on a non-interactive connection before closing it. This coupled with the default maximum number of connections at 100 quickly leads to "no connections" errors when using persistent database connections in php on multiple database systems.

The eZ team seem to be focusing on decreasing the resource requirements for the next version. Tuning the operating platform (database web server and OS) to your particular needs is an important and ongoing task to ensure you get the best out of your resources.

I hope that you get to the bottom of this and that you let us all know what the root of the problem is.

Cheers
Bruce

Tuesday 20 April 2004 1:34:13 pm

Hi Zinistry,

eZ publish deletes all cache blocks when any content is published, as you probably know. This would cause the slowdown when you have many editors working. One way to improve performance in this situation is to use the "ignore_content_expiry" parameter in your cache blocks.

http://ez.no/ez_publish/documentation/incoming/cache_block_optimization
(see "Expiry"blunk.gif Emoticon

The more complex your templates are (pagelayout, menus etc.) the more this will help. When the content of the cache blocks actually needs to change, you must clear the cache blocks "manually" (using "Setup" -> "Cache" in admin). But you probably don't need to do this often, only when the menu content and similar parts should change.

Tuesday 20 April 2004 3:46:46 pm

Hi!
So, if I use this "ignore_content_expiry", and the editors publish their content, will the content then imediately be updated for the users?

Why empty all the cache blocks when the content is beeing published? Why not just empty the content for THAT page?

Every time when i make changes in templates, I stop apache, empty the cache from Turck accellerator, and clear the cache with ./bin/shell/clearcache.sh --clear-all, and then starts apache again. I guess I don't have to do any more after making changes to the ez publish template system?

There is also a bug in the clearcache.sh, because almost every time when I use this, lot of content ..or sometimes just content on few pages are gone..we only see blank pages..and then I have to run the script again, and sometime 3-4 times, then all the content is available. Pretty weird.

Today our site went down 2 times again. The other websites on the same server were still up & running, but not ez publish. The eZ team is now going to check out the system, find out what's making the problems, and do some changes to speed up stuff, so I hope we all can get some good tips and hints out of this.

I also think 3.4 will make things better happy.gif Emoticon

Modified on Tuesday 20 April 2004 3:48:05 pm by K259

Tuesday 20 April 2004 4:01:59 pm

Hi,

-> clearcache.sh / blank pages

I think, that problem occurs because ez don't use the file-locking at the cache-section, as I mentioned somewhere ago (bug reports / forums).

I've the same problem if I delete the whole cache and many visits are at the same time on my sites. So, if the cache is empty and two instances tries to write / append the same cache-file, some weird results can happen.

So, the only way around is:
Delete the cache more then once and pray that not much visitors are coming at the moment.

Kind regards,
Emil.

Tuesday 20 April 2004 4:04:31 pm

Zinistry:

3.4 will help: If http://ez.no/community/blog/pre_caching_of_view_cache works then you could set cache expiry to manual and turn on previewcache. This will mean new page are the only ones updated.

Please post back any hints and tips. They will be thankfully received.

-- tony

Modified on Tuesday 20 April 2004 4:05:46 pm by Tony Wood

Tuesday 20 April 2004 4:06:11 pm

Ok, since other eZ guys are working on your site, I will not go into specifics. Just a few comments:

You said: "Why empty all the cache blocks when the content is beeing published? Why not just empty the content for THAT page?"

Because the published object sometimes influences the appearance of the cache blocks. How can eZ publish know which cache blocks contains code that will be influenced by the change in the published object? You would need an advanced artificial intelligence to check that.

You said: "So, if I use this "ignore_content_expiry", and the editors publish their content, will the content then imediately be updated for the users?"

Cache blocks where "ignore_content_expiry" is used will NOT be updated on publishing. They will only be updated when you clear them manually, as in my previous comment. This is good when you have complex pieces of template code that are rarely influenced by content publishing, or template pieces that do not need to be continuously updated (you could clear cache blocks once per day, for instance).

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from