eZ Community » Forums » General » Image Gallery upload makes heavy load...
expandshrink

Image Gallery upload makes heavy load over mysql

Image Gallery upload makes heavy load over mysql

Thursday 03 November 2011 6:46:58 pm - 12 replies

Im trying to find out how to reduce the load over mysql when someone uploads images. Everytime they are uploading inmages (one by one or using multiupload, doesnt matter) the server will have very high mysql load and all site will slow down.

 

We are using GD not ImageMagik, could this be an issue or where can we dig to find the solution (or maybe a bug).

 

Thank you.

Thursday 03 November 2011 6:58:49 pm

Hello Luis,

Sorry I don't know more to suggest on the areas involved.

I could be wrong here, but I really don't think there is a performance difference (related to the issue you describe) between using GD and ImageMagik.

I also wonder if this article (while older) might help

http://share.ez.no/learn/ez-publish/tuning-mysql-for-ez-publish/(page)/5

 

I hope this helps ...

 

Cheers,

Heath

Modified on Thursday 03 November 2011 8:14:59 pm by // Heath

Friday 04 November 2011 9:39:23 pm

As Heath said, there should be no big perf issue between using GD vs ImageMagick. It looks more like an issue when the system is trying to store the content object after the image has been uploaded.

Questions:

  1. is it also slow when someone creates another object? Let say an 'Article' or a 'Frontpage'.
  2. is your eZ Publish installed on a single server or on a cluster.
  3. if on a cluster are you using eZDFS or eZDB?
  4. how big is you current eZ database?

Hint:

Try to enable the SQL debugging in your site.ini.append.php and tail the logs.

Friday 11 November 2011 4:06:32 am

NGUYEN,

Thanks for your help.

OK so performance is not a difference for GD vs IM... so in what is better IM? happy.gif Emoticon

 

About the questions :

 

1. Not that slow... I mean I can see in Munin how the mySQL server will go 150% up when they are loading pictures, but content is created almost every 3 minutes with no issues at all.

2. We now have it in a cluster: web+mysql

3. None of them in fact I dont know them what are they for? is it worth it?

4. DB should be around 2GB and 15,000 objects... we are in the process of making EZFind take care of Searching...

 

Thanks for your time.

Friday 11 November 2011 4:09:02 am

Hello Luis,

Sorry I don't know more to suggest on the areas involved.

I could be wrong here, but I really don't think there is a performance difference (related to the issue you describe) between using GD and ImageMagik.

I also wonder if this article (while older) might help

http://share.ez.no/learn/ez-publish/tuning-mysql-for-ez-publish/(page)/5

 

I hope this helps ...

 

Cheers,

Heath

Heath, I read the article and we are implementing those recomendations hope they help... Also I appreciate your comment about GD vs IM.I will come back with comments.

Friday 11 November 2011 4:57:15 am

Hello Luis,

I took some time tonight to write some of my thoughts on your question about the differences between the available open source image libraries. For me I see a lot of performance differences, feature differences, license differences and a lot more that's harder to summarize. But here are my thoughts regardless ...

 

A brief comparison of GD, ImageMagick and GraphicsMagick

I have heard GD is good and in some use cases faster than ImageMagick. GD according to wikipedia (gd site is not fully online currently?) uses a bsd like license [9]

But then again I heard even more people saying ImageMagick in a lot more ways is even better suited for the task and in some situations better results across the board. Though I've heard still others that preferred GD2's quality in other use cases.

In the past I have heard some folks were put off (to say the least) by the license choice of ImageMagick (Their own, I don't have a link to it, more on this in a second). Today I was fixing a MacOS MacPorts problem (latest 2 versions of ImageMagick for macports segfault for one of our clients local dev, had to roll back 2 versions to get it going again). When I was doing this work and research into ImageMagick and Macports I noticed that they had a recent commit changing (in macport packages at least) from their own license to the Apache-2 license. I can't even begin to share how excited I got to read that commit [0] I think this change will bring even more growth for this project. I know I sure respect them for this a lot more today than yesterday!

Now on to GraphicsMagick, seems to have been based on ImageMagick [5] with differences[1][3][4][8] I'm sure, on of them being the distribution under the MIT license [2], which some users preferred over the ImageMagick's previous license [6]. License aside it seems a lot of folks say GraphicsMagick is faster in some use cases[4].

I think in the end I stick with ImageMagick myself for most general use cases especially since it is now distributed under the Apache-2 license. Though in performance critical systems I will prolly seek to leverage the fastest tool possible for the job this prolly means GD or GraphicsMagick. Nice thing is they all (now) use licenses which I enjoy using and would have no problem relying upon.

 

References

[0] https://trac.macports.org/changeset/85397/trunk/dports/graphics/ImageMagick/Portfile

[1] http://www.graphicsmagick.org/benchmarks.html

[2] http://www.graphicsmagick.org/Copyright.html

[3] http://stackoverflow.com/questions/862051/imagemagick-vs-graphicsmagick

[4] http://www.admon.org/graphicsmagick-vs-imagemagick/

[5] http://lwn.net/Articles/26637/

[6] http://www.imagemagick.org/script/license.php

[7] http://jamesarmes.com/blog/2009/03/php-image-benchmarks-gd-vs-image-magick

[8] http://stackoverflow.com/questions/5282072/gd-vs-imagemagick-vs-gmagick-for-jpg

[9] http://en.wikipedia.org/wiki/GD_Graphics_Library

 

I hope this helps ...

 

Cheers,

Heath

Modified on Friday 11 November 2011 5:01:53 am by // Heath

Saturday 14 April 2012 5:00:12 pm

Heath,

 

Im sort of 6 months late here! Not receiving update emails from this site sad.gif Emoticon

First of all thank you for taking all that time to elaborate. I read and looked at all links and I can see that GD is somehow marginally or a lot (as the final link says) perfmormer than IM... but seems that IM is not the best at all... so my question would be: why EZ uses IM as default? hum...

I have been trying to find out how to sort out with this problem I still have. I still have problems uploading photo galleries... what seems to happen is: The picture is upladed, then it gets uncompressed then cropped and finally compressed... this done by maybe 20 or 30 pictures will simply bring down the performacne dramatically but this performance degradations seems to happen AFTER the upload. I mean exactly that... I upload pictures and everything seems to go fine pictures are publsihed... then maybe after aprox. one minute or sort of, the system will go slow for as long as 20 minutes with multiple PHP processes running. Then it will suddenly come back to normal. Checked logs and everything seems to be fine...

I've been having this issue now for months so we decided to remove multiupload and downgrade picture quality, that seems to help a lot.

I am still using GD and I really think I will keep on with that after reading you investigation!!

 

Thanks a lot.

Saturday 14 April 2012 5:01:24 pm

Quote from Quoc-Huy NGUYEN DINH :

As Heath said, there should be no big perf issue between using GD vs ImageMagick. It looks more like an issue when the system is trying to store the content object after the image has been uploaded.

Questions:

  1. is it also slow when someone creates another object? Let say an 'Article' or a 'Frontpage'.
  2. is your eZ Publish installed on a single server or on a cluster.
  3. if on a cluster are you using eZDFS or eZDB?
  4. how big is you current eZ database?

Hint:

Try to enable the SQL debugging in your site.ini.append.php and tail the logs.

Hi Huy!

I know its been 6 months (ups!) but did you had the time to read my reply?

Saturday 14 April 2012 5:24:02 pm

Quote from // Heath :

Hello Luis,

I took some time tonight to write some of my thoughts on your question about the differences between the available open source image libraries. For me I see a lot of performance differences, feature differences, license differences and a lot more that's harder to summarize. But here are my thoughts regardless ...

 

A brief comparison of GD, ImageMagick and GraphicsMagick

I have heard GD is good and in some use cases faster than ImageMagick. GD according to wikipedia (gd site is not fully online currently?) uses a bsd like license [9]

But then again I heard even more people saying ImageMagick in a lot more ways is even better suited for the task and in some situations better results across the board. Though I've heard still others that preferred GD2's quality in other use cases.

In the past I have heard some folks were put off (to say the least) by the license choice of ImageMagick (Their own, I don't have a link to it, more on this in a second). Today I was fixing a MacOS MacPorts problem (latest 2 versions of ImageMagick for macports segfault for one of our clients local dev, had to roll back 2 versions to get it going again). When I was doing this work and research into ImageMagick and Macports I noticed that they had a recent commit changing (in macport packages at least) from their own license to the Apache-2 license. I can't even begin to share how excited I got to read that commit [0] I think this change will bring even more growth for this project. I know I sure respect them for this a lot more today than yesterday!

Now on to GraphicsMagick, seems to have been based on ImageMagick [5] with differences[1][3][4][8] I'm sure, on of them being the distribution under the MIT license [2], which some users preferred over the ImageMagick's previous license [6]. License aside it seems a lot of folks say GraphicsMagick is faster in some use cases[4].

I think in the end I stick with ImageMagick myself for most general use cases especially since it is now distributed under the Apache-2 license. Though in performance critical systems I will prolly seek to leverage the fastest tool possible for the job this prolly means GD or GraphicsMagick. Nice thing is they all (now) use licenses which I enjoy using and would have no problem relying upon.

 

References

[0] https://trac.macports.org/changeset/85397/trunk/dports/graphics/ImageMagick/Portfile

[1] http://www.graphicsmagick.org/benchmarks.html

[2] http://www.graphicsmagick.org/Copyright.html

[3] http://stackoverflow.com/questions/862051/imagemagick-vs-graphicsmagick

[4] http://www.admon.org/graphicsmagick-vs-imagemagick/

[5] http://lwn.net/Articles/26637/

[6] http://www.imagemagick.org/script/license.php

[7] http://jamesarmes.com/blog/2009/03/php-image-benchmarks-gd-vs-image-magick

[8] http://stackoverflow.com/questions/5282072/gd-vs-imagemagick-vs-gmagick-for-jpg

[9] http://en.wikipedia.org/wiki/GD_Graphics_Library

 

I hope this helps ...

 

Cheers,

Heath

Heath,

Noticed in this link http://sven.webiny.com/php-gd-vs-imagemagick-benchmark/ that memory performance seems to be better in IM. I dont care if it is 0.4 sec slower... so I will give IM a try...

Saturday 14 April 2012 5:58:15 pm

Hello Luis,

Good to hear from you again.

I like to think that eZ uses IM by default because it is already available in most lamp stacks and generally accepted already (license and performance issues aside).

The issue based on your description

Remember image alias variation images are not created on upload rather on first use within a browser so this is much less an issue (durring upload). My point being that the system will by default slow down in exactly the way you describe as it has to create the image alias image variations for all the images uploaded and browsed on the user siteaccess. This is slow and based on user actions after upload (variable rate). You could use a tool like bcimagealias to generate image alias variation images on image upload (and beyond) in an effort to pregenerate the files before they are requested by the user's browser. https://github.com/brookinsconsulting/bcimagealias

This might help improve user page / image content loading as it is already ready (cached)

You might also want to try to increase the resources available to the system in an attempt to increase performance and responsiveness.

I think multiupload can put a temporary strain on a service by nature of the solutions at hand. Use with caution. 

I hope this helps ...

 

Cheers,

Heath

Sunday 15 April 2012 8:52:01 pm

My take on the Imagemagick vs. GD discussion:

1. you mention "db going to 150%" when uploading images, but it is not very clear to me what you mean by this: is that the cpu of the db server? If so, is that server separate from the webserver?

2. cpu usage of the dbserver should be completely unrelated to image processing. Regrdless of gd/imagemagick or else, if cpu usage of the db goes through the roof it is because of some other thing going on. Maybe content indexation? Did you try enabling delayed indexing?

3. a big advantage of imagemagick over GD is that it executes in a separate process from php. This is good because you then do not need to set a huge memory_limit in php.ini, and is friendly to Apache memory usage on high-raffic servers

Friday 11 May 2012 9:28:23 pm

Quote from Gaetano Giunta :

My take on the Imagemagick vs. GD discussion:

1. you mention "db going to 150%" when uploading images, but it is not very clear to me what you mean by this: is that the cpu of the db server? If so, is that server separate from the webserver?

2. cpu usage of the dbserver should be completely unrelated to image processing. Regrdless of gd/imagemagick or else, if cpu usage of the db goes through the roof it is because of some other thing going on. Maybe content indexation? Did you try enabling delayed indexing?

3. a big advantage of imagemagick over GD is that it executes in a separate process from php. This is good because you then do not need to set a huge memory_limit in php.ini, and is friendly to Apache memory usage on high-raffic servers

Gaetano:

Thanks for taking the time.

1. Yes we have separate servers so we could manage this.. we have around 240,000 daily pageviews on this server, and 600 avg. concurrents.

2.In fact after your post I have noticed that, not only is mySQL having those peaks when generating new content (first it was images, now even with articles) but we are now also having web server peaks at the same time and plenty of PHP processes that will remain running for as long as 10 minutes. I would like to check that delayed indexation... where should I look for it? is it in sites.ini?

3. Yes! thats what I have been thinking about. I have managed to finally get it to work but still have it disabled, I will give it a try this next week and will come back with results.

Friday 11 May 2012 9:32:19 pm

Quote from // Heath :

Hello Luis,

Good to hear from you again.

I like to think that eZ uses IM by default because it is already available in most lamp stacks and generally accepted already (license and performance issues aside).

The issue based on your description

Remember image alias variation images are not created on upload rather on first use within a browser so this is much less an issue (durring upload). My point being that the system will by default slow down in exactly the way you describe as it has to create the image alias image variations for all the images uploaded and browsed on the user siteaccess. This is slow and based on user actions after upload (variable rate). You could use a tool like bcimagealias to generate image alias variation images on image upload (and beyond) in an effort to pregenerate the files before they are requested by the user's browser. https://github.com/brookinsconsulting/bcimagealias

This might help improve user page / image content loading as it is already ready (cached)

You might also want to try to increase the resources available to the system in an attempt to increase performance and responsiveness.

I think multiupload can put a temporary strain on a service by nature of the solutions at hand. Use with caution. 

I hope this helps ...

 

Cheers,

Heath

Hi Heath!

I will definetly have this done after checking basic configs... like delayed indexing and some other tasks... But will take a look at this so we may optimize this as possible...

I have EZ 4.3... are there any updates you are aware of that may improve performance?

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu

Proudly Developed with from