eZ Community » Forums » General » Faster cache clearing?
expandshrink

Faster cache clearing?

Faster cache clearing?

Tuesday 25 October 2011 11:52:36 am - 10 replies

Hi all,

On some of the larger sites we run we're having issues with the long time it takes to clear cache. Whenever we deploy something we need to clear the cache to ensure changes appear. Our automated deploy process does something along the lines of:

  • Put up a maintenance page
  • Download changes from Git
  • Clear cache
  • Automatically warm up caches
  • Remove maintenance page

The cache clearing step can take considerably amount of time. It currently takes about 20 minutes and during those 20 minutes the site is down. The site runs in a cluster using eZDFS with 2 web servers. 

One idea we had was to just rename the cache folder which can then be cleaned up at a later point and entires in ezdfsfile would be deleted using ezcache.php. This worked great on 4.3, however on 4.4 running ezcache.php --clear-all --purge does not remove all cache entires in ezdfsfile. This causes all sorts of weird behaviour as there's now references to files in ezdfsfile which no longer exists on disk.

Does anyone have any ideas on how to speed up the clear cache process? 

One solution that could help is if eZDFS stored cache files in a separate table from images and binaries. That way you could just rename the cache folder and truncate the cache table and you're done. This would probably reduce the 20 minutes it takes for us down to a few seconds tops.

 

Cheers,
Ole M.

Modified on Tuesday 25 October 2011 5:02:56 pm by Ole Morten Halvorsen

Tuesday 25 October 2011 8:18:34 pm

I was going to suggest exactly the same thing that you did.

This is a known problem, that makes also backups harder in ezdb/ezdfs configurations, and makes master/slave setups slower than they should be (if slave is used only for HA purposes). Anyone willing to submit a pull request for this change is very much welcome!

Tuesday 25 October 2011 8:19:08 pm

ps: in the meantime, maybe you can experiment with table partitioning...

Wednesday 26 October 2011 11:39:24 am

Cool, thanks for the input Gaetano. I'll see if I can get a pull request together happy.gif Emoticon

Wednesday 26 October 2011 3:19:18 pm

Looking through the DFS code this might be slightly trickier than I thought. The current implementation has no concept of file types (image vs other files vs cache files). 

My current thinking is to introduce an optional parameter in eZClusterFileHandler::instance() to specify the file type. This type will then be passed on to eZDFSFileHandler and eZDFSFileHandler will pass this type on to the backend implementation which can then use it to determine if it's a cache file and if yes, use the cache table instead of ezdfsfile.

This approach should be backwards compatible. If you do not update cache generating code to use the new second parameter it will just store it in ezdfsfile and a cache clear will purge it anyways.

The other approach which I think would be a lot dirtier is to infer the file type using the file path. 

Any thoughts?

 

Ole M.

Wednesday 26 October 2011 4:29:26 pm

Adding an extra parameter forces to change all the code calling the functions that manipulate cache/storage data. Which is maybe good in longterm, but hard to apply to all existing extensions (there is a "scope" paraemeter in there somewhere which I think was meant for this but never used very much). And we will have a new storage layer anyway at some point in the future...

I'd just go the dirty way and use regexps to decide. Maybe storing the list of regexps that validate a file as a cache file in an ini - or having some smart way to detect user-added caches anyway.

Wednesday 26 October 2011 5:46:53 pm

There is in the ezdfsfile database datatype and scope, when I look into,

I see for datatype for example:

  • audio/mpeg
  • image/jpeg
  • video/x-flv
  • xml
  • php
  • misc

I see for scope for example:

  • binaryfile
  • classattridentifiers
  • classidentifiers
  • content
  • designbases
  • expirycache
  • image
  • media
  • mediafile
  • rsscache
  • sortkey
  • statelimitations
  • template-block
  • user-info-cache
  • wildcard-cache-index

so it looks like, you can identify the caches with scope.

 

Greetings, ekke

Modified on Wednesday 26 October 2011 5:47:31 pm by Ekkehard Dörre

Thursday 27 October 2011 10:00:11 am

There is in the ezdfsfile database datatype and scope, when I look into,

I see for datatype for example:

  • audio/mpeg
  • image/jpeg
  • video/x-flv
  • xml
  • php
  • misc

I see for scope for example:

  • binaryfile
  • classattridentifiers
  • classidentifiers
  • content
  • designbases
  • expirycache
  • image
  • media
  • mediafile
  • rsscache
  • sortkey
  • statelimitations
  • template-block
  • user-info-cache
  • wildcard-cache-index

so it looks like, you can identify the caches with scope.

 

Greetings, ekke

Hi Ekke,

There's a scope, however there's lots of methods that just take a file name. E.g.:

 $clusterHandler->fileExists(  'var/site/cache/ezcontentlanguage_cache.php' );  

Modified on Thursday 27 October 2011 11:21:47 am by Ole Morten Halvorsen

Friday 28 October 2011 4:26:43 pm

Added pull request:

https://github.com/ezsystems/ezpublish/pull/152

Saturday 29 October 2011 2:56:03 pm

Added pull request:

https://github.com/ezsystems/ezpublish/pull/152

Great, thanks a lot, looks very good,

 

Greetings, ekke

Monday 31 October 2011 5:10:07 pm

Excellent!

Lots of memories are coming back from reading this post big-smile.gif Emoticon

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from