eZ Community » Forums » eZ Publish 5 Platform » slow publishing in production
expandshrink

slow publishing in production

slow publishing in production

Wednesday 04 March 2015 4:42:42 pm - 14 replies

Hi,

In prod environment, publishing a content is very slow and occured often a timeout.

In test and dev environments, with the same datas, it's quick.

With the debug, I can see that "node cleanup" takes long time.

Do you know what is this problem ?

Thank you in advance for your help.

CH

Saturday 07 March 2015 1:58:56 pm

Hard to tell.
I had actually never noticed this specific accumulator before after working on and around eZ Publish for some years :P But a quick case insensitive search I found that it is Node Cleanup/"node_cleanup" in ezcontentcachemanager.php around line 765 depending on version.

It might be because of viewcache cleanup, so 1. do you have other viewcache.ini rules? 2. might just plainly be because you of course have a lot more cache files in prod, is this cluster (dfs/nfs)?

It might also be one of the events listening to this (this was added in one of the 5.x versions to let Platform stack listen to cache clearing events), so to rule that out you can apply something like this:
https://gist.github.com/andrerom/60a33e662e300e05e3ba

Anyway, I'm afraid you need to do some debugging in prod, make sure to test patches you apply in staging first as this is class is used a lot.

Modified on Saturday 07 March 2015 1:59:58 pm by André R

Tuesday 10 March 2015 2:29:14 pm

Slow server response or maybe the internet connection

Wednesday 11 March 2015 2:28:13 pm

Hi,

I'm working with Céline on this problem.

 

1. do you have other viewcache.ini rules?

Yes, we tried with     SmartCacheClear=enabled, but it's the same result

 

 

2. might just plainly be because you of course have a lot more cache files in prod, is this cluster (dfs/nfs)?

The cache used is the default one: ezfsfilehandler

But we have 2 servers pointing to a unique storage (nfs) for the website sources. Both servers write into this storage with the user (www-data), and same rigths.

 

 

We added the debug accumulator to the code (https://gist.github.com/andrerom/60a33e662e300e05e3ba) and we have a new ligne on the debugger with the impacted nodes count (2) and the % of duration (~ >50)

 

 

 

Slow server response or maybe the internet connection

I don't think so, because the problem appears only after a certain amount of time whitout any content publication. If we publish a second content just after, the publication is much faster.

 

We compared the number of files deleted on the folder .../ezpublish/cache/dev/stash when publishing

Before: folder=845 & files=784  --   After  folders=284 & files=220

 

In production environnement, we have more than 20000 folders and 10000 files.

The time to delete all these files often causes a timeout  (again, after a certain amount of time whitout any content publication)

 

 

Any ideas ?

Thursday 12 March 2015 9:11:11 pm

Hi,we had this problem on a high load ez5 server as well.One issue was that the backend worked with a wrong setting (viewcaching=off). This causes not just performance problems during publish but other problems as well: eZ tries to remove stash spi cache entries recursivly which leads tsometimes to timeouts.Another tip is to configure stash using memcache.

Saturday 14 March 2015 10:47:05 am

Hello there

"But we have 2 servers pointing to a unique storage (nfs) for the website sources. Both servers write into this storage with the user (www-data), and same rigths."

Unfortunately, it is very possible that this causes most of your issues. Is the var folder shared over NFS like the source itself ?

This is a setup we do not support. The basic reason is simple: cache clearing operations (in legacy) rely on files mtime. NFS is not meant as a real time file sharing system. It has a local cache on each mount point, and this local cache keeps changes for up to 30 or 60 seconds. The article Use of NFS considered harmful covers the topic in details.

First, the source code should probably not be shared this way. It causes more harm than good, and can quite easily be avoided by using deployment tools. But the source isn't the biggest issue, var is. The var folder should *always* be local to each installation, and not shared.

Sharing of cache/storage data, in var, is covered in Legacy by what we call Clustering. The "latest" implementation, DFS, uses NFS to store binary data, but uses MySQL to handle metadata (modification time, etc), and optimizes IO operations on NFS. It is used on large installations, and is meant exactly for this case.

Note that depending on how much you use the new stack, there are further considerations. Legacy clustering is compatible with the new stack, but Symfony offers much more possibilities when it comes to high performance and redundancy.

Monday 16 March 2015 1:15:18 pm

Hi Bertrand,

i think the problem also occurs when stash cache is stored locally. 

When a lot of content is edited and requested the stash cache grows. After a few days publishing content slows down since eZ tries to remove to many cache objects stored in stash cache. We are using still filesystem as storage engine for stash. 

Some times max execution time is reached (30 seconds). The timeout occurs in the stash removal method (recursive removal). We are using the standard smart view cache settings in legacy. The site itself is build in Symfony2 / eZ 5 only. It use around 16 languages. Stash cache is around 1.5 GB.

The problem also occurs when you try to change roles or content classes in the backend (legacy)

Wednesday 18 March 2015 3:01:47 pm

Hi Bertrand,
And thanks! You were right, it's because of NFS.
We'll envisage to use memcache instead of filesystem.
But our website has lots of contents, so we think that it could be used too much memory.
We wonder if it'd be safe to use memcache in this context. What do you think about it? And have you a recommendation for memory requirements?
Thank you in advance for your help.
CH

Tuesday 24 March 2015 12:22:34 am

Hi Céline,

The Persistence cache is for rather small data structures, and memcached can be configured to remove stale/old cache, so I would not worry so much about memory use with it. If eZ Publish is setup in cluster it is the only way to go, or be experimental and use Redis which is also supported by "Stash".
But with NFS now out of the picture(it should be, no matter if you use memcache it is NEVER a good solution unless you use it as Bertrand mentioned, together with DFS Clustering), is there any issues left? If not I would skip it for now, add it when you add more than one server.

Thursday 20 August 2015 5:27:23 pm

Hello everyone,I'm facing the same problem Frank here.Our site is also working in Symfony2 / eZ5 and publishing time is gonint to over 3 min.We added many logs inside all publishing methods and it really seams like cache problem.The object is published in admin (eZ Legacy), and then eZ goes to Symfony to clear cache, and this process takes long time.The ezpublish.yml is set to use FileSystem, and inMemory = true.We couldn't find an exit to this problem yet. Céline says to use "memcahe". Does it need to be in a cluster instalation? Our isn't...

Thursday 20 August 2015 9:41:17 pm

Hi Alexandre,we have solved this issue by using a memcache server for storing stash data. you can configure eZ for using stash and memcache. You do not need a cluster setup. The problem are recursive remove operations when you publish content. Using memcache this will be no problem at all.Good luck

Friday 28 August 2015 3:07:04 pm

Hello Frank, thanks for your reply!

Memcache did "most of the job" here. All publishing are working fine now.

But one other issue related to publishing came up, the draft saving. Basically eZ5 comes with auto-save for drafts activated, and this process still going into loop (recursive remove operations).

We disabled the auto-save for drafts and in a straight through publishing everything goes fine. But if the user try to save the draft the site goes to the loop and stays there until apache times out (5 min). Strange thing is that after apache times out the site continues normally, as like nothing happened.

Our version of eZ is 5.4.0. We found this documentation talking about "Transparent caching principle referred to above has one major obstacle".

See here: https://doc.ez.no/display/EZP/Persistence+cache#Persistencecache-Legacykernelcachepurging

Does anyone have any idea about how to solve this?

Friday 28 August 2015 8:51:02 pm

Hi Alexandre,

you are welcome. 

I am wondering why saving a draft should trigger the cache system. Because it is just a draft. We don't have such problems in out 5.4 installations. 

There is just a problem with especially autosaving if you e.g. edit a file containing large attachements. eZ is generating new versions and the attachments are copied but not removed later on. We posted a bugreport and it will be fixed soon. 

Wednesday 09 September 2015 5:47:54 pm

Hi Frank, thanks again!

It's good to know we can count on eZ community! happy.gif Emoticon

Could you send us the bug report ID so we can keep track too?

Wednesday 09 September 2015 8:29:12 pm

Hi Alexandre,

this is the issue i mentioned:

https://github.com/ezsystems/ezpublish-legacy/pull/1192

In our case it happens during autosave. 

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from