I love a good challenge. Aside from teaching others how to succeed and surpass my knowledge, scalability and performance is probably what I’m most passionate about. During my time with EllisLab I’ve had the fortune to work with developers of some of the largest ExpressionEngine-based websites. And sometimes even the best of developer runs into performance issues. The more I help people with their performance issues the more I notice it’s the same few things causing performance issues. The following is an outline of how I like to approach troubleshooting, along with some tips and tricks to consider as you are developing and deploying your ExpressionEngine-based websites.
I realize this is really long, and there’s a ton to digest. We’ll be creating a new section in the ExpressionEngine User Guide where these tips are broken apart for easier consumption.
Great article, Greg! I will refactor some conditionals based on your advice.
I knew that it isn’t the best approach to use too much embeds, but the snippets can’t be saved as files, so I think that’s the reason many developers tend to use embeds instead of snippets, maybe in a next update EllisLab can help us with that?
That’s a tough one, ultimately, it comes down to performance. I think it was Erik Reagan who blogged a while back (I’m not finding the link for the life of me) about how he saw performance increases when not saving his templates as files. If Erik didn’t author that post, I hope someone will correct me. This is because, as it stands now, ExpressionEngine is querying the database to figure out what template it should use, then looking on the file system. If the template doesn’t exist on the file system, we’ve just had an unnecessary disk request that depending on server strain can contribute to raising disk IO activity.
It makes so much sense why you’d want to be able to save them as flat files, but the idea behind snippets it to keep them FAST. So really, I see it as a tradeoff that is well worth it. Make sense?
Do you have any suggestions in ways we might be able to make the experience with snippets a bit more convenient for you? If so, please let us know.
I would like to be able to save snippets as files during development. When the site is done, I synchronize the files and disable the “save templates as files” option (that was mentioned in a EE podcast #36 and maybe in the forum too).
Usually when I need a relative complex snippet, I code it in Espresso and then I copy and paste the code in the EE CP, I would like to avoid that step.
If the system is on and the snippets are files, you can put a warning somewhere.
I had the unfortunate experience of too much traffic, a problem we should all have, this year!
To say that we tweaked would be an understatement!
We were running on a dual core 3.0 Pentium and serving up to 100,000 page views a day in EE…mostly forums. Obviously the traffic peaked in certain hours of the day…as usual, so it really was at a hourly level where we hit the wall - maybe at 7,000 page views per hour. Again, mostly forum, which require a lot of queries and the db is fairly large (almost a gig)......
I was able to keep thing barely alive - server loads of 10 or less and page processing times of 8 or so second (WAY too long, IMHO, but fine in a pinch) using every trick in the book. I offloaded some images, banners, css and js to Amazon, tuned mysql, etc. etc…...
However, in the end the KISS way to deal with heavy loads - IMHO - turned out to be hardware! Yeah, lots of people will pooh-pooh that as bad form. But all the stuff we did (and my ISP knows his stuff in Mysql and Apache) did was make thing bearable, not fast….and fast is what you need to succeed today.
We replaced the older server with a quad-core Xeon 2.5. and increased RAM from 2 to 4G.
At first glance, a lot of folks would think that would help, but not make all the difference in the world….
Well, I’m here to report to you that it DID make all the difference in the world. We replaced the server during a time of heavy traffic - and since it has been replaced I have not seen a page load time (as EE reports) of over 1/2 a second. Most are approx. 0.3 second. Of course, that is just a report back from the db side of EE, but according to google webmaster tools our total web serving speed went from as high as 7+ seconds average to 2.5.
So while all the coding excites me, in the end the best solution is the one that takes the less time and money and creates the best result. Call me quick and dirty, but I’m all for throwing hardware at a problem…ram and cpus are cheap compared to coding time!
I would like to be able to save snippets as files during development. When the site is done, I synchronize the files and disable the “save templates as files” option (that was mentioned in a EE podcast #36 and maybe in the forum too).
Usually when I need a relative complex snippet, I code it in Espresso and then I copy and paste the code in the EE CP, I would like to avoid that step.
If the system is on and the snippets are files, you can put a warning somewhere.
From a workflow standpoint, this makes a lot of sense, as you get more flexibility when you develop and more performance in your production environment. I’ll make sure the rest of the team sees this! Thanks!
We replaced the older server with a quad-core Xeon 2.5. and increased RAM from 2 to 4G.
At first glance, a lot of folks would think that would help, but not make all the difference in the world….
Well, I’m here to report to you that it DID make all the difference in the world. We replaced the server during a time of heavy traffic - and since it has been replaced I have not seen a page load time (as EE reports) of over 1/2 a second. Most are approx. 0.3 second. Of course, that is just a report back from the db side of EE, but according to google webmaster tools our total web serving speed went from as high as 7+ seconds average to 2.5.
Definitely. Sprucing up to more modern hardware with a bit more RAM from a P3 would have a large, immediate impact!
So while all the coding excites me, in the end the best solution is the one that takes the less time and money and creates the best result. Call me quick and dirty, but I’m all for throwing hardware at a problem…ram and cpus are cheap compared to coding time!
But this is a means to an end, don’t you think? Tossing hardware might be a good short term solution, but you’re never improving the underlying issues that caused the problems in the first place. If you’re to a point that you have a cluster of 4-5 web servers, with quad core CPUs and 8 gigs of ram, offloading to a CDN, what happens when your traffic spikes next? Toss more gear at it?
If anything, I’d take the calm times after upping that gear to really dig in and get these issues solved. Worst case, you can maybe downgrade ram, shut off a virtual and keep it on standby as a spare in case you get Fireball’d. Best case, you fix the issues and you and your sysadmins can sleep a bit more easy. :D
Fantastic article, and although I have already run into all of these issues (and resolved them) while implementing ExpressionEngine in enterprise environments I believe the items covered here are helpful to most anyone who is concerned with performance.
I did find that a lot of scenarios did not lend themselves to being able to be streamline or tuned in any way that would make them process or load faster. Sometimes certain templates are always going to be heavy, no matter how well you tune ExpressionEngine, Apache or MySQL. My work around for stuff like this (and this is recommended in the forums somewhere, just can’t remember where) is to make good use of jQuery and Ajax. Pages that can be looked at as a series of components or widgets can often be easily configured to be Ajax-ed. I created a lot of self-contained stub templates that had one purpose and one output, and on pages that required multiple stubs I would utilize ajax loading for individual components. This allows your page to complete loading much faster and the user(s) or visitor(s) get a better user experience overall. This also makes the debugging of slow loading components much easier.
So on the heals of a great article, that’s my 2 cents.
Bjørn,
Perusing your code leads me to believe there is an additional library required for it to work. Is that accurate? Would you mind updated your repo with it? I’m interested in checking it out
If I understand the part about multiple if’s versus the if elseif structure correctly, you’re saying that in terms of template parsing, single {if}{/if} blocks get parsed near the beginning, while {if}{if:elseif}{/if} blocks get parsed near the end?
Currently working a large EE site. I never want to really dive into performance and optimization until I have that feeling that the EE programming is complete. Like decorating a cake before adding the finishing touches.
I sent this to a few coworkers and the ultimate question we had is why the EE platform parses advance conditionals last?
I’m trying to work out how to maximize the website performance.
In order to do that I decided to do a breakdown (timing) of a typical HTTP request.
It takes 821ms (819ms waiting, 2ms receiving) to load the test page (Google Chrome Developer Tools):
- Script processing takes 242ms (EE Display Output Profiler, Total Execution Time); Apache claims it takes 300ms to serve the page (The time taken to serve the request)
- Server ping takes 100ms
Any idea of what is happening during the remaining 421ms (821 - 300 - 100)?!
To give you an idea:
- When saved as a flat html file, the same test page takes 112ms to load
- Website Bloomberg.com has 75ms ping - and it takes 77ms to load the homepage
If you’re to a point that you have a cluster of 4-5 web servers, with quad core CPUs and 8 gigs of ram, offloading to a CDN, what happens when your traffic spikes next? Toss more gear at it?
-greg
Since I assume we are in the top 10 or 20 EE powered sites (at least the known ones), the percentage of sites which would hit this wall would be so low as not to concern any average developer…......which is where my practicality comes in!
The new server is nowhere near top of the line - hardly even mid-line, and yet could handle (IMHO) over a million visitors per month of normal use.
The very few who would have more traffic than that have a lot of headroom in terms of faster single servers…then perhaps dual servers (mysql and other)......and then, it would seem, we are to the amount of sites that you could count on your fingers (in terms of actual real world EE sites).....
It seems to me the bigger problem with EE performance is the vast number of people who host them on very low cost programs…using 1/200 of a server.
Maybe that list needs officially refuted….if we want to add some additional validity to the idea that a single server cannot handle 99% plus of EE sites…...??
Why aren’t anyone talking about caching? That is, for me, one of the most crucial parts for getting a snappy site. I did some benchmarking and got a 90x increase in possible pages/second: http://expressionengine.com/forums/viewthread/186140/
Without running every request through PHP, it feels very fast, and our mediocre VPS server have no problems at all.
Why aren’t anyone talking about caching? That is, for me, one of the most crucial parts for getting a snappy site. I did some benchmarking and got a 90x increase in possible pages/second: http://expressionengine.com/forums/viewthread/186140/
Without running every request through PHP, it feels very fast, and our mediocre VPS server have no problems at all.
I think most apache and ‘nix setups automatically cache php and even mysql (has built in cache) ...at least that is what my admin told me! This requires a decent amount of RAM, but memory is cheap these days.
