We are needing support with an EE install that is consuming a ton of resources. Specifically CPU.

We have the database and web files on a dedicated server running Nginx instead of Apache.

Server specs are Quad Core Woodcrest Xeon 2.33 Ghz with 8GB of memory.

The website is for a daily newspaper and on high traffic days is getting about 22,000 unique visitors and 110,000 page views. About 40% of the site’s traffic comes in a span of 4 hours in the mornings.

MySQL is using a ton of CPU and grinding the site to a halt during busy times. We have tried everything we can think of to get things back to normal and nothing is fixing the problem.

Any suggestion, ideas, anything would be most appreciated.

Is there someone you can suggest who knows EE and can be hired to review our server set up?

Anna

yikes!

Do you have access to server logs?  There might be something in the apache error log to tell you what’s causing this.

What are your settings at:  CP Home ›  Admin ›  System Preferences ›  Database Settings

-greg

Database Connection Type: Non Persistent
Enable SQL Query Caching: Yes

———-

Yes, we have full ssh access and can see all log files.

Our errors are in the Nginx logs and we’re seeing this a lot:
2008/10/06 11:55:33 [error] 19991#0: *165353 upstream timed out (110: Connection timed out) while reading response header from upstream

There are no errors in the MySQL logs.

Anna

We are also seeing this in the log file:

2008/10/06 07:24:15 [crit] 30618#0: *212356 open() “/etc/nginx/html/50x.html” failed (24: Too many open files)

have you asked the host as to why that beast of a machine is getting brought down to it’s knees?  If so, what kind of feedback did they give you?

More background on the hosting…

We were previously running the site on two VPS, one running apache and php and the other running MySQL.

After two months of stability, three weeks ago we started having problems with the server setup.

After not being able to stabilize the server, we moved it to this dedicated server this weekend. Today was our first day of heavy traffic.

The host normally sets up Apache on their servers, but custom installed Nginx as a test to see if it used less resources then Apache while running EE.

So they are looking at things right now but since it’s a dedicated server we are ultimately responsible for getting things working.

The database is about 260MB and it will only get bigger.

—————-

Why do you think MySQL would be using so much CPU?

At 260MB, is our database large for an EE database?

Is this an EE issue or a server issue?

Can you share server info for the EE websites so I can get some perspective? What kind of server, what kind of page loads, what kind of unique visitors, what size in the database?

Does EE work with a server administrator or do you guys do everything in house? If you work with someone, is that person available for hire? If not, do you have someone to recommend who can look at our server setup?

Anna

EngineHosting.com, Anna.  The best of the best - and the would absolutely be able to help you with a solution on their servers.

What version and build of EE are you on, and are you running any extensions?  The amount of traffic you’re having is not really that high.

Hi Lisa,

EE INSTALL INFO
ExpressionEngine 1.6.3
Build:  20080421

EXTENSIONS
Category Checkboxes (v.1.0.0)
Bad Behavior 2 Extension (v.2.0.0)
Edit Alert (v.1.0)
Checkbox (v.1.0.1)
Clone Entries (v.1.1)
Comment Spam Prevention (v.1.1)
Required Category Extension (v.1.0.3)
Edit Remember (v.1.1.0)
Edit Tab AJAX (v.1.2.2)
Field Header (v.1.0.1)
Field Section (v.1.0.1)
Filter By Author (v.1.0)
Hidden Weblogs (v.1.0.3)
LG Add Sitename (v.1.0.1)
LG Member List (v.1.2.1)
LG Polls: Controller (v.1.0.6)
LG Polls: Poll Question (v.1.0.6)
File, by Mark Huot (v.3.1.1)
Multi Text (v.110)
onSubmit (v.1.0.1)
Publish Page Improvements (v.2.0.0)
Tab Forward (v.1.0)

Does Engine Hosting do managed dedicated servers?

I need you to disable all extensions and see if your CPU usage goes down.  I know that your site is likely reliant on them but you have a lot of extensions, some with histories of problems, and possibly some extension conflicts.  We need to see if you experience the speed issue with un-extended ExpressionEngine.

Also, 1.6.4 was a recommended security and maintenance release - you really need to update to 1.6.4.

Ah, yes, that’s actually the only kind they have, I believe.  See EngineHosting Solutions: Managed Dedicated Servers.

I disabled all extensions and if anything it seemed like CPU went up. Not sure if that was really true.

I watched the CPU using top for about a minute then had to turn the extensions back on.

Are there extensions you know have problems. If they aren’t needed, I can disable them permanently.

Is upgrading to 1.6.4 something you need me to do to troubleshoot this problem right now or can I do it at a later time?

When you’ve seen high mysql CPU usage, has it been an EE issue or a server issue?

—————————

Here is what I’m seeing in top for CPU:
Cpu0 : 50.7% us, 26.0% sy,  0.0% ni, 23.3% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu1 : 48.5% us, 23.3% sy,  0.0% ni, 28.2% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu2 : 44.2% us, 18.9% sy,  0.0% ni, 36.9% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu3 : 49.8% us, 19.7% sy,  0.0% ni, 30.4% id,  0.0% wa,  0.0% hi,  0.0% si

Here are the processes:
19845 mysql   15   0 527m 75m 4772 S 161 0.8 110:54.20 mysqld
22509 nobody   16   0 31404 12m 3372 S   32 0.1   1:09.99 php-cgi
23583 nobody   16   0 31508 12m 3376 R   31 0.1   0:48.29 php-cgi
23617 nobody   16   0 29424 10m 2808 S   17 0.1   0:23.34 php-cgi
23618 nobody   16   0 27904 9696 3372 R   17 0.1   0:12.79 php-cgi
22513 nobody   16   0 28656 10m 3396 S   14 0.1   1:05.70 php-cgi

Anna

We’ve seen this for several reasons - database connections not dropping (shouldn’t be an issue with non-persistent); spam referrers/bots/comments/trackbacks, unoptimized servers, badly coded templates, and extension problems or conflicts between extensions.

The best test would be to install a completely clean copy of ExpressionEngine and see if it experiences the same problems.  That would be the fastest and easiest reduction test.  Are you able to try that?

Are you saying a second copy of EE on the server connecting to a second database or to the live database?

A second database; a totally stand alone installation as a pure test.  But this would need to be done with your current site off to be a real test, or we could have cross-contamination.

I can’t turn the live site off.

Well, we have a bit of a problem here. We need a situation whereby we can reduce and isolate the problem.  This problem could stem from a ton of possibilities, as listed above.  You’re running a ton of extensions, including ones with a known historical problem such as the File extension, you may have template optimization issues (we haven’t even gone there yet), and you’re running a non-typical web server configuration using PHP as CGI.  Your environment may not be perfectly tuned for a dynamic database-driven system as well.

What we can do right now without touching your site is look at your templates (You can post one, such as your index, and we can have a quick look), ask your server administrator more about those errors in the server error log and why they’re occurring, and see where we get from there.

You can also make a complete backup of your site and put it on a development environment and see if you have speed problems, and do some more intensive testing there since it won’t affect the live site.

An option, of course, is to move to a known, excellent, hosting environment with administrators that are skilled and knowledgeable in tuning servers for high traffic, dynamic sites.  That would take out a major, possible contributing factor.

We have the two old VPS servers that were hosting the server before I moved it to the dedicated server yesterday. One is running PHP and Apache and the other is running MySQL. We can test with that.

The errors are being looked at too by the host.

Great, let us know what your test comes up with on those VPS servers, please!