eZ Community » Forums » Setup & design » eZ is DOSing server, need help
expandshrink

eZ is DOSing server, need help

eZ is DOSing server, need help

Wednesday 12 January 2005 9:48:56 pm - 19 replies

Hello!

Yesterday I've setup my site running eZ installations, suddently eZ started to DOSing server, CPU usage went 100%, Apache "Max Servers" limit was also acquired.

I've enabled all caches / template compiling etc:

TemplateCache=enabled
TemplateCompile=enabled
TemplateCompression=enabled
Debug=disabled
ViewCaching=enabled

.htaccess:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\.(gif|jpe?g|png|css|js|htm?|pdf|swf)$
RewriteRule ^(.*)$ /index.php/$1 [L,QSA]

I've tested performance with ab utility, with -c 10 -n 10:
Percentage of the requests served within a certain time (ms)
50% 2113
66% 2445
75% 3211
80% 3865
90% 3865
95% 3865
98% 3865
99% 3865
100% 3865 (longest request)

Percentage of the requests served within a certain time (ms)
50% 676
66% 938
75% 938
80% 938
90% 1315
95% 1315
98% 1315
99% 1315
100% 1315 (longest request)

with -c 25 -n 25:

Percentage of the requests served within a certain time (ms)
50% 4952
66% 6390
75% 6697
80% 7719
90% 27715
95% 27811
98% 27836
99% 27836
100% 27836 (last request)

Percentage of the requests served within a certain time (ms)
50% 12233
66% 15263
75% 19928
80% 22600
90% 53653
95% 56540
98% 68262
99% 68262
100% 68262 (last request)

After this server is almost DOSed :/

System specs:
Dual 2.4ghz Xeon w/1GB of memory, Redhat ES v3.

What should I do? Any hints?

Wednesday 12 January 2005 10:45:05 pm

Does your server swap at all? Try o reduce the swap used ...

The optimum is that you do not swap at all to reduce load from the disk IO

other steps

- install a accelerator e.g phpa
- clear cache

Wednesday 12 January 2005 11:16:28 pm

Nope, server still has 300MB of free memory, cache is cleared, PHP accelerator is enabled.

Thursday 13 January 2005 12:10:38 am

Hi!
Must say that Marek is a very modest person.

We need help with the website of Polish Mozilla Community - http://mozillapl.org While testing - it worked quite well.. but it only needed a official blast-off, couple of hours and traffic at ~375 000 page views/ day to make our server go offline sad.gif Emoticon

We're one of the Biggest Open Source communities in Poland. We tried everything we could but there's not even a SINGLE person within us that knows eZ moze than Marek. So if he can't help all our HOPE is in YOU.

PS
My english is not as good as I supposed sad.gif Emoticon Sorry

Modified on Thursday 13 January 2005 12:11:46 am by Michal Dyro

Thursday 13 January 2005 12:45:35 am

Debugging is always hard when you don't have access to a system. So how about some more information?

php version ?
apache version ?
ez publish version ?

The systems seems to have worked in testing, so what has changed since going live?

> CPU usage went 100%
So what process is taking up all the CPU?

> Apache "Max Servers" limit was also acquired.

This could be an apache issue. It could be caused by the system getting many hits and grinding to a halt. The hits continue coming but the apache processes are unable to finish serving the requests due to high server load.

Can you access the site at all? How long does it take to become unusable?

Cheers
Bruce

Thursday 13 January 2005 1:18:57 am

More information:
PHP 4.3.10
MySQL 4.1
Apache 1.3
eZ 3.5

Since testing only eZ directory has changed, eZ was moved to root directory.

Adminstrator told that apache processes are eating CPU, eZ couldn't send a page and all requests were hold with "sending reply" status... When DOSed I can't access the site, apache must be restarted...

Thursday 13 January 2005 1:33:16 am

You told us you moved the server. Does that mean you actually had a non eZ website running there before?

If this is the case try to redirect all requests that belong to a non eZ url to an error page ( e.g everything *.html )

In those cases eZ will give a module not found error those error have a high impact on the system they ea a lot of cpu time.

Besides that you may have reached a critcal limit in request per server. It might be just the case that his maschine can't handle it with eZ running on it. I have never been on a SINGLE server that can serve more then 15 eZ pages per second.

In this case block all networking traffic and do a benchmark with no traffic and compare it with your usual traffic.

Modified on Thursday 13 January 2005 1:43:33 am by Björn Dieding@xrow.de

Thursday 13 January 2005 1:39:11 am

An other idea...

have you checked the mysql? When mysql has problems it might be that mysql requests are set to a hold state... during that time apache just hold it till it timeouts...

Usually in this senario apach and mysql will eat up all the ram and swap... i guess this is not the case

Thursday 13 January 2005 1:41:04 am

Is debugging disabled? It should... You want as less IO as possible

Thursday 13 January 2005 9:08:49 am

If you don't already use it, get turckmm cache. It greatly speeds up large PHP Apps like eZPublish by caching the bytecode.

I just saw you use PHPaccellerator. That should do the job, too.

Modified on Thursday 13 January 2005 9:18:01 am by Gabriel Ambuehl

Thursday 13 January 2005 3:17:46 pm

have you checked the mysql? W
hen mysql has problems it might be that mysql requests are set to a hold state... 
during that time apache just hold it till it timeouts...

This maybe a reason. We had problem with server DOSing by eZ publish when mysql crashes. CPU usage went to 100%. Accessing main page takes very very long time ...

Modified on Thursday 13 January 2005 3:18:37 pm by Łukasz Serwatka

Thursday 13 January 2005 4:50:39 pm

1. Yes, main page (website) was using Postnuke CMS
2. I agree that some non existing URL could be redirected to eZ, but this doesn’t explain that eZ is also DOSing when moved to other directory and tested with ab utility
3. Site is not running on dedicated server, but I assume that if it could handle Postnuke with 1%-2% load it would handle eZ
4. Site is using eAccelerator (new mmcache)
5. I don’t think that this is MySQL issue (will ask administrator), only one request is made when page is cached

Some more info on my settings:

site.ini override:

[Session]
SessionNameHandler=custom

[DatabaseSettings]
DatabaseImplementation=ezmysql
Charset=utf-8
Socket=disabled
SQLOutput=disabled

[ContentSettings]
ViewCaching=enabled
PreViewCache=disabled
PreCacheSiteaccessArray[]=mozillapl

[TemplateSettings]
TemplateCache=enabled
TemplateCompile=enabled
TemplateCompression=enabled
Debug=disabled

[DebugSettings]
DebugOutput=disabled
DebugRedirection=disabled

site.ini for siteaccess:

[SiteSettings]
ErrorHandler=defaultpage

[SiteAccessSettings]
RequireUserLogin=false
ShowHiddenNodes=false

[SiteAccessRules]
Rules[]=Access;disable
Rules[]=Module;user/register

[RoleSettings]
EnableCaching=true
PolicyOmitList[]
PolicyOmitList[]=content/pdf
PolicyOmitList[]=content/read
PolicyOmitList[]=form
PolicyOmitList[]=rss/feed

Note: Some settings were removed

Thursday 13 January 2005 5:13:22 pm

I think you should let a professional look at your setup and test it. I like said before it is very hard to resolve from the outside. Root access is required too.

Please check also the disk IO, is this also running at 100% or is it just the CPU?

Thursday 13 January 2005 5:17:05 pm

Waht happens if you disable those 2 options?

[ContentSettings]
ViewCaching=enabled

[TemplateSettings]
TemplateCache=enabled
TemplateCompile=disabled
TemplateCompression=disabled

Thursday 13 January 2005 6:02:29 pm

Made some more test, seems that this is caching issue with disk access, with following settings:

[ContentSettings]
ViewCaching=enabled
PreViewCache=disabled

[TemplateSettings]
TemplateCache=disabled
TemplateCompile=enabled
TemplateCompression=disabled
Debug=disabled

I can run ab with -c 50, of course reply times are not so good (longest reply 11 sec) but it isn't DOSing the server... Do you have any ideas on improving eZ caching system? Any info about internal aspects of TemplateCache?

Friday 14 January 2005 8:29:04 am

Are you able to actually get a page from the server?

Have you tried pre compiling the templates?
http://ez.no/community/forum/install_configuration/zz_systems_or_cpu_100

Have you tried disabling template compiling?

Cheers
Bruce

Friday 14 January 2005 9:25:26 am

There is not much more you can do to improve the caching.

An option is to make static copy of the whole site. So that you don't hit mod_php4.

What disk is actually in the server? I think providing a better and faster disk will help you a lot.

Friday 14 January 2005 2:26:39 pm

I had a similar issue with 3.5 and an upgraded php 4.3.10 with apache2. (we run apache1 on the production server, never tried to see if the problem existed there)

Having run out of things to try, I decided to go the 'old school programmer route'.. randomly commenting out bits of code. After disabling phpa, and the load problems went away. I suspect it had something to do with an older version of phpa not liking the newer php version and apache2.

Friday 14 January 2005 6:12:50 pm

Server is running only IDE disks, even not in RAID... But managed to make some static cache system... Mayby we will improve that and send it to contributions happy.gif Emoticon Thanks for help!

Sunday 16 January 2005 9:40:10 pm

From a unix sysadmin point of view it would be interesting to see the output of the unix command "top" which should should cpu state and which processes specfically are using the processor/memory as well as state of those processes.
In addition, I would check the error/message logs for apache and mysql respectively and ensure thats its not an issue being reported by either one.
Another thing to check is any email to users running crontab entries for ezpublish (eg runcronjobs.php etc).

ie just determine that it is actually ezpublish causing the problem by asking questions like, if no users log in to ezpublish does the load increase anyway?

hope thats of some help
Arran

expandshrink

You must be logged in to post messages in this topic!

36 542 Users on board!

Forums menu