Thread

We are running:
lighttpd/1.5.0-r2533, PHP 5.2.10 (fastcgi), MySQL 5.0.45, CentOS 5.3, ExpressionEngine v 1.6.8

We run off of 4 servers:
3 x dual quadcores with 12GB of ram - (1 db, 1 web, 1 cgi)
1 x single quadcore with 4GB of ram - (cgi)

All servers are synced to an iscsi, so that there is no bottle neck with our harddrives, we also have all of our images running off of a CDN.

All of our servers are very healthy and no where near max load limits, yet our site completely chokes when we approach 5,000 - 6,000 concurrent online users.

PHP appears to be processing close to 60-80 requests/second until that point, at which things slow down to 4-5 requests/second, which makes no sense.

We believe that the bottleneck lies with ee as we also run a copy of Invision power board on the server, and it is blazing fast, even during the times our ee site is choking.

We are having an extremely difficult time figuring out what the issue maybe and would greatly appreciate some assistance on this matter.  Is ee bottlenecked at a certain point?  Is there anything we can do to free this bottleneck?

jaydee,

What version and build of EE are you using?

If using build 20081212 and above you can avail of some newer traffic features which could help you out in the area of concurrent users.

EE Docs : Handling Extreme Traffic with ExpressionEngine

EE Blog :Traffic Management Features in the 20081212 Build

We are on the latest version and build.
ExpressionEngine v 1.6.8
Build:  20090723

We’ve already looked through all of the documentation and utilized all of the resources available.  But none of that is helping, as our servers are perfectly healthy. ee just seems to hit a brick wall at a certain point, which is a complete mystery to us.

We might possibly be running the most active and highly trafficked ee site on the internet, as we can’t find anyone experiencing the same symptoms as us, and we are struggling to find a solution.

jaydee,

We have quite a few high traffic sites running on EE without problems so lets examine what you have going on here.

Can you let us know what options you have enabled regarding the extreme traffic features? Use screen-shots if necessary.

What kind of caching have you enabled in EE?

The issue may even be with your templates and how their setup is approached. Do you use a lot of embeds or are you using related entries for example?

Do you know how many queries a typical page produces? {total_queries}

Do you see anything in your mysql slow query log?

Would you be able to provide a link to the site? You can email us if you don’t feel comfortable posting it here

I can honestly say I have seen more than one site push beyond 15k online users before needing to enable the settings outlined in the extreme traffic handling doc section.  In fact some of those sites were part of the reason those settings were added into 1.6.6+ rather than needing to comment out core file lines.  But every site, and the features and functions used are unique of course, but again issues rarely, if ever, don’t show as heavy resource usage in the process.

But as EE simply is a web application relying on http server (apache, etc.), php and mysql to execute code and supply it data, if EE was being pushed to a form of “limit” it would appear in high load usage for the underlying technologies being used, as EE can’t hide a bottleneck outside of visible server resource usage.

A quick question would be do you pull in any outside data feeds into EE add-ons, or php in templates, ie. XML feeds, API calls to outside hosted services, etc?  If those outside services are slow to respond, you will see php execution pause waiting for timeouts.  Its a common, but generally overlooked issue that can impact sites, but generally you would see a backlog of long running http requests building up too.  Just a quick thought to go along with John’s existing questions.