Thread

I am currently using 1.6.9 ExpressionEngine.

I have not installed any new plugins of late, nor have I have changed the system setup in any way recently, such as changing templates or adjusting caching functions.

For the past few weeks, out of the blue, my hosting service had been “randomly” suspending my site without warning because my site had “apparently” caused sever overload. Sadly, the hosting company was unable to give me any further diagnostic info except for saying that it was ExpressionEngine that was causing the problem (but without saying what within the application was the culprit).

I was given an excerpt of the server log that showed the abrupt sudden increase in server load, which I am posting below (the username & directory had been changed for privacy reasons):

15:13:15 up 16 days, 16:34, 1 user, load average: 40.75, 41.22, 24.30

Running Proceses: XXXXX 4758 0.0 0.0 3420 532 ? SN
14:40 0:00 /usr/lib/courier-imap/bin/imapd/home/XXXXX/mail/mysite.com/XXXXX/

XXXXX 13386 0.0 0.0 3424 464 ? SN 14:54 0:00 /usr/lib/courier-imap/bin/imapd/home/XXXXX/mail/mysite.com/XXXXX 
XXXXX 21918 0.1 0.3 39460 12696 ? DN 15:07 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 21985 0.0 0.3 38692 12432 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22014 0.0 0.2 38920 12084 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22125 0.0 0.3 39432 12980 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22142 0.0 0.3 38680 12404 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22253 0.0 0.3 38664 12388 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22254 0.1 0.3 38664 12368 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22267 0.0 0.3 38680 12404 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22286 0.0 0.3 39952 13496 ? DN 15:08 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22297 0.0 0.3 38664 12392 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22311 0.0 0.3 38692 12396 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22325 0.1 0.3 40016 13844 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22434 0.1 0.3 40016 13936 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22437 0.0 0.3 38692 12440 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22441 0.1 0.2 38692 11760 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22506 0.1 0.2 38924 12224 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 22560 0.1 0.2 38664 11788 ? DN 15:09 0:00 /usr/bin/php /home/XXXXX/public_html/index.php 
XXXXX 24751 30.0 0.4 39736 19224 ? RN 15:13 0:00 /usr/bin/php /home/XXXXX/public_html/index.php

As shown above, the load increase occurred abruptly, and the only clue I had was it was related to index.php, which in itself was entirely unhelpful.

The problem I am having now is that this occurred too rarely for me to troubleshoot by changing settings in the site, such as turning on/off caching for certain templates or removing plugins/add-ons, etc. It simply would be too difficult to correlate any observed changes to confirm or refute that that a particular setting was the cause.

One thing I have excluded is that the server load was not related to a sudden increase in web traffic. In fact, according to the traffic data collected in real-time by another application on that server during these incidents, the number of visitors and hits were nearly identical.

Has anyone seen this previously with 1.x ExpressionEngine, even in older builds? I had not tried to “optimize” the database in myPHPadmin, as I don’t believe this would fix the problem. I was told by my hosting service that it was the ExpressionEngine software and not the server that was the problem, though I have serious doubts about that claim.

Can someone please offer me some suggestions?

Sadly, the hosting company was unable to give me any further diagnostic info except for saying that it was ExpressionEngine that was causing the problem (but without saying what within the application was the culprit).

This might just be your site being hit by traffic, or perhaps spammers. Are you using the blacklist?

15:13:15 up 16 days, 16:34, 1 user, load average: 40.75, 41.22, 24.30

What am I looking for here, exactly? The load (40+!) does look quite high, but that’s hardly your site alone.

I was told those numbers represent server loads over the past 5, 10, 15 minutes (normally it should be less than 2; since the server my site is on has 2 CPU cores). The line states the load was from 1 user, and apparently it was my site (yikes!).

Yes, I am using the blacklist function within EE, which I understand are written to .htacces (which I can see0.

The problem I have with troubleshooting is that any function used by EE to render the page will always be attributed to index.php, so knowing that server load was high with running the index.php is not helpful in finding out the source of the problem.

Sadly, the hosting company was unable to give me any further diagnostic info except for saying that it was ExpressionEngine that was causing the problem (but without saying what within the application was the culprit).

This might just be your site being hit by traffic, or perhaps spammers. Are you using the blacklist?

15:13:15 up 16 days, 16:34, 1 user, load average: 40.75, 41.22, 24.30

What am I looking for here, exactly? The load (40+!) does look quite high, but that’s hardly your site alone.

P.J., are you using the Referrers module? If so, are you getting any sort of hits there? How about your 404 page? Have you set up one, or do 404s go to your index.php?

Yes, I am using the referrer module. Currently, I have only turned on weblog entry view tracking. The other trackings were turned off (including referrer tracking). The max number to save is set at 500. Having this said, the referrer log is empty (since I never turned it on).

My 404 page is actually set up using .htaccess. It is linked to a static .shtm outside of ExpressionEngine that I made and just uploaded to the server. The 404 page is a separate page and does not redirect back to the index.php. Under “access” option for each template, I simply left the URL setting to default for “unauthorized access”.

Your latter two questions are interesting ones. Are there hidden places where I might have inadvertently redirecting invalid/404 requests back to the home index.php?

P.J., are you using the Referrers module? If so, are you getting any sort of hits there? How about your 404 page? Have you set up one, or do 404s go to your index.php?

P.J.,

1. What is set for your 404 Page under Templates ›  Global Template Preferences?
2. Do you have any PHP in your templates, many embedded templates, or many exp:weblog:entries tags on one template?

This may be a case where you’ll need to use the process of elimination to narrow down the issue. I’d start with this page and incrementally add caching to see if there’s any change.

1. It is set at weblog/errors.
2. Yes, some of the pages have PHP codes in addition. However, these pages had been entirely remained untouched for months (if not longer). The server load issue was never encountered until recently and occurred rather abruptly.

As I said in my earlier reply, the “process” of stepwise elimination is not practical to carry out. This is because I cannot make a change and then wait (for unknown periods of time) to see if my hosting service would shut my site down. Rather, I need a list of mostly likely candidates, make changes to ALL of them at once, and then see if these would fix the issue.

Unfortunately, we don’t offer that level of support. However Solspace does have a service that might help.

http://www.solspace.com/blog/entry/solspace_now_offering_performance_evaluations_for_expressionengine_sites1/

I am certain that my site is not close to the level of traffic that required the level of performance evaluation solspace is offering.

Instead, I am simply looking for troubleshooting tips elsewhere within EE that is causing this server overload.

Somehow, this conversation is being distracted by the impression that this is due to a high-traffic issue. I assure you this is NOT the case, as the site is independently monitored by awstats which shows basically the same level of traffic.

For example,

Will it help if I optimized the EE sql databases using phpmyadmin?
Will it help if I trim the number of old templates to keep from 10 (which is the current level) to 5?
Will it help if I set the 404 Page under Templates ›  Global Template Preferences to none?
Will it help if I set the error page to static, assuming somehow that page is being rendered a lot?

Hi P.J.,

The first thing I’d try is setting the 404 Page under Templates ›  Global Template Preferences to None, as it sounds like you have two 404 handlers in effect - I’m not sure what the ramifications of this are but it’s probably not helping.

If you’re still being suspended, refer to the page I recommended earlier. You can certainly implement those recommendations at one time if you’d like.

That type of a load probably signifies more than a simple problem. Some points….

1. A load of up to about 8 is usually acceptable - at peak times. My server has hit 10, but starts slowing down at 8 or so.
2. There are a number of things you can do to tune up apache and mysql, but I suspect your problem is elsewhere.

I would look at the server logs LIVE when the load increases like that and see if something looks suspicious. Last time I had a problem like this, it was related to an old PHP or CGI script (form) which I had left on my server (outside of EE), which had been found by spammers and was being hit every second…...so it was obvious from the apache logs.

Another time, my admin guy found that I had somehow left a mysql port (of some type) open, so that maybe mysql was being hit from somewhere else.

Processes can also get hung up…...a good restart of apache and mysql should show you if this is the case…

Stuff like this is not too hard to work through if you use the process of elimination! Check all the various logs - the message logs, the apache logs, etc.- something is up…..

The line states the load was from 1 user, and apparently it was my site (yikes!).

No, that one user does not mean the number of people using the site! That just represents the one UNIX admin logged in or the fact that the server processes are running under one username!

I have 300 people on my server now - and see the attachment? one user…...

Usually, it is up to the admins to help you (support) find the source of the problem. As I suggested, I doubt that it is traffic. Something is getting hit….like a stray script or open port…..

Thanks for the assist, Craig.

PJ, does the additional info help?

Yes, Sue, please keep thread open for now. I am in communication with my hosting service about the suggestions made here.

Hi P.J.,

Will do, please let us know what they say.

Cheers

Greg