Thread

I am in the process of doing a site that will get several thousand people a day. Does ExpressionEngine slow down with big sites and can it handle them well? Is it memory intensive on the server?

Thanks

There is a good overview in the docs.  Data Caching and Performance

There are also a lot of third party add-ons which can help with caching and performance.  The community driven resource for add-ons is Devot://ee.

Server setups aside, how well your site will perform largely depends on how it was put together.  There are ways that you can shoot yourself in the foot if you are new to ExpressionEngine, but you will be fine if you read through the docs and follow the guidelines.

I suggest turning on template debugging while you work so that you can see what the templates are doing and how much work they are having to do.  If you feel that a template is having to work too hard, then you can make tweaks to optimize it.

If you are totally new to ExpressionEngine and after you have built your site you decide that it’s too slow, then you could have another ExpressionEngine developer take a look at your templates to point out where you might have gone wrong.

Welcome to the ExpressionEngine forums, davidwhite.

Thanks for the assist, John!

As John mentioned, a lot will depend on how you set things up, and on where you choose to host. If you’re going to be having a lot of traffic, you’ll want to consider using a quality host. That can make all the difference.

Does the information from John help?

EE can certainly slow down like any other CMS. The key like John wrote is tuning it and building out your platform to support it. Backend caching can be a big help to create truly static pages, we use litespeed HTTP. Also, concise code and optimized images will help along with offloading static content to a CDN when possible. Memory usage depends on your sites functionality, for instance if you are working with a lot of image manipulation (resizing with plugins and such) then your memory will definitely be greater than something more static. All-in-all with just a few thousand visitors per day you shouldn’t have any huge bottlenecks as long as you don’t go crazy with your templates.

Very good points all around.

Some of the most trafficked sites on the Internet are powered by ExpressionEngine, so it can definitely handle the load. It’s all a matter of using proper coding techniques (like not nesting Channel Entries tags inside another pair of Channel Entries tags), a powerful web host, and a well-tuned caching system. For sites that are getting big with a lot of traffic, I’d recommend taking a look at a caching system like Varnish.

Is there anything else we can help with?

Kevin,
We have been doing research into caching systems and are seriously considering Varnish. Since you mentioned it here, do you have any helpful resources you can point us to on implementing varnish along with EE. We have already read though the great Varnish/EE post by Kevin Cupp. Do you know of any others?

Hi Adam,

It seems there are a limited number of other resources when it comes to using Varnish and ExpressionEngine. A couple of case studies and a podcast that might prove informative.

Cheers!

Thanks Dan. Well, we are setting up a test environment as we speak and I will for sure try to share our findings and experiences with the community.

Hey Adam,

Fantastic! Keep us posted on your progress!

Cheers!

I am in the process of doing a site that will get several thousand people a day. Does ExpressionEngine slow down with big sites and can it handle them well? Is it memory intensive on the server?

Thanks

We serve 20,000 people a day with relative ease - on a quad-core dedicated server with 4G ram. My guess is that we could double that with almost no delay in serving pages.

You can work backwards from there for a rough rule of thumb. That is, if you can get a powerful shared server where you are one of 10 accounts on the server, that should do the job nicely.

However, if you just go for the lowest price service available, they often put hundreds on the same server!

If you are expecting several thousand folks a day and using EE (paid) CMS, you should invest in the best host you can realistically afford. This goes for most CMS software since they all make a lot of calls to the database.

My guess is that shared, VPS and/or low cost dedicated servers and other setups which you could rely on would cost $50 to $150 a month.

All just estimates…of course! Other factors come into play such as the size of the EE total content, etc.

Following up on Adam’s post, we have now successfully deployed a Varnish cache as a front end to our ExpressionEngine installation. Performance improvements are *astounding*, to say the least. Under normal load, our servers (we have two separate machines, one running Apache, one MySQL—varnish is also running on the former) run at about 1/3rd (or less) the load they did previously. When we’re having a traffic event, load pretty much doesn’t move.

I’ve pasted our our varnish VCL config below for anyone who’s interested, but in essence, we’re caching pages for users who are NOT logged in, caching images and CSS for everyone, and keeping those caches for 60 seconds (that magic being explained in #4 below). Yes, we could go longer, but doing it this way allows us to simply “plug-and-play.” Even if an article is updated it will never be more than 1 minute old for the users. In addition, we do a few things

1) we pass along HTTP_X_FORWARDED_FOR with the actual client’s IP. This is helpful for both logging and for any operations for logged-in users that require their actual IP.

2) We blacklist several of our URLs that we *know* should never be cached (our back end, our store, our Premium Mac Geek Gab—since it requires a login, etc).

3) We whitelist a few things that should be cached REGARDLESS of whether a user is logged in (image, CSS, and Javascript files, our home page, and one other script).

4) THE KEY: the test for the presence of the exp_uniqueid cookie is how we’re deciding whether or not someone is logged in. If it exists, then ALL cookies are passed through (resulting in the page being read from Apache, not Varnish’s cache), if it DOESN’T exist, then NO cookies are passed through, allowing varnish to deliver a result from the cache if one exists.

The only remaining issue (that we can see, anyway 😉), is that we need to figure out what to do with IP-based blacklists, etc., for security reasons. Obviously EE is no longer seeing each user on the site (if they aren’t logged in) and isn’t seeing their direct IP as REMOTE_ADDR if they are, so… there’s still some work to do. But by and large it’s working phenomenally well, and my only regret is not having done this the day we launched with EE years ago. We’re back to the performance we had with static HTML pages, and that’s a welcome return.

-Dave

# Configure backend
backend default {
    .host = "[IP of Apache server]";
    .port = "[port of Apache Server]";
    #Set first_byte_timeout so varnish waits longer than 60 seconds for long responses, like large publishing operations
    .first_byte_timeout = 180s;

sub vcl_recv {

    # Pass along client information to backend
    remove req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = client.ip;
    
    # Don't cache EE system directory, special directories, ACT requests or POST requests
    if (req.url ~ "^/[ee_backend_url_here]" ||
 req.url ~ "^/tmo/store" ||
 req.url ~ "^/tmo/podcast/premium" ||
        req.url ~ "ACT=" ||
        req.request == "POST") {
        return (pass);
    }

    # Always cache the following file types for all users
    # making sure that =css and .css are both properly cached since we use min
    # and ^/$ matches the home page, which we'll probably want to remove when we add logins there.
    if (req.url ~ "(?i)(=|\.)(png|gif|jpeg|jpg|ico|swf|css|js)((\?[a-z0-9]+)|(&[a-z0-9]+))?$" ||
 req.url ~ "^/$") {
     unset req.http.Cookie;
    }

    # Test for presence of login with cookie exp_uniqueid. If present, pass through ALL cookies.
    # If not present, pass through NONE. This effectively decides whether or not
    # a request is cached. If exp_uniqueid exists, not cached.
    if (req.http.Cookie ~ "(?i)exp_uniqueid") {
 return (pass);
    }
    else {
 remove req.http.Cookie;
    }
     
    set req.grace = 1h;
    
    return (lookup);
}

sub vcl_fetch {

    set beresp.grace = 1h;
    
    # We set it to cache at 60 seconds for non-logged in users. That way even if we update an article it's still going
    # to be no more than 1 minute old for anyone.
    set beresp.ttl = 60s;
    
    return (deliver);
}

I kept the above post here because it was a part of this conversation, but if the mods feel it should be moved to its own thread, by all means, please do. 😉

Hey Dave!

I think the nature of this thread has taken on a life of its own. With that, I’m going to move it in to General Discussion to keep it going and for posterity 😊

Thanks everyone for your input!

Cheers!

Sounds good. I’m just hoping that folks searching for Varnish and ExpressionEngine will find it, that’s all. 😉 It was pretty easy to set up, but would have been even easier with an EE-specific example like this one.

i ‘m agree with Kevin. and also if site is well managed from the start of development, there will n’t be slow processing.