Troubleshooting Site Performance Issues
I love a good challenge. Aside from teaching others how to succeed and surpass my knowledge, scalability and performance is probably what I’m most passionate about. During my time with EllisLab I’ve had the fortune to work with developers of some of the largest ExpressionEngine-based websites. And sometimes even the best of developer runs into performance issues. The more I help people with their performance issues the more I notice it’s the same few things causing performance issues. The following is an outline of how I like to approach troubleshooting, along with some tips and tricks to consider as you are developing and deploying your ExpressionEngine-based websites.
I realize this is really long, and there’s a ton to digest. We’ll be creating a new section in the ExpressionEngine User Guide where these tips are broken apart for easier consumption.
Output Profiler
The output profiler and template debugging options in ExpressionEngine are, in my opinion one of the most powerful features we have at our disposal. Everything you need to diagnose performance issues is right there in your browser. You don’t need to be a sysadmin or know a thing about how your web server works in order to see what’s going on. In fact, as a sysadmin I rarely, if ever shell into a system while troubleshooting because you could never get this information from the webservers. However, I do highly recommend everyone have some sysadmin-foo, just so you understand the tireless work they perform.
The Super Admin should make it a habit to never turn off Display Output Profiler and Display Template Debugging. These tools provide a wealth of information on what exactly ExpressionEngine is doing to render the page.
In ExpressionEngine 2.x, the Output Profiler displays query times along with the queries being executed. With proper MySQL caches and buffers in place, an ideal query time should be sub 0.02 seconds on channel entries queries, however most should typically be sub 0.00 seconds. Carefully study query times.
Template Debugging shows the amount of time it takes to process a template, as well as the total processing time, and memory usage. Here, look for unusually large jumps in time. If you see a large jump or spike this is a bottle neck, and you should review your templates to see if you can alleviate it.
Templates
Templates are where most performance issues start. Since ExpressionEngine provides the developer with near infinite control over the display of their markup and code, the possibilities are endless. When developing your ExpressionEngine-based websites with template debugging and Show MySQL queries on, please keep the following in mind.
Nesting Tags
As a general rule, never nest tags such as the channel tag within a channel tag, or a channel tag within a categories tag. The inner tag will be run and parsed on every iteration through the outer tag, and depending on the number of results returned from the outer tag, queries can add up in a hurry.
Advanced Conditionals
Placing ‘heavy duty’ tags such as channel entries/categories tags within advanced conditionals will lead to unnecessary overhead on both the web server and database server. As per ExpressionEngine’s Template Engine’s Parse Order, Advanced Conditionals are among the last things parsed. This means tags such as channel/categories entries on the templates are fully parsed and removed from the final template at the last moment.
If you find yourself with a template like:
{if segment_2 == 'test'}
{exp:channel:entries channel="test"}
// Code here
{/exp:channel:entries}
{if:elseif segment_2 == 'another'}
{exp:channel:entries channel="another"}
// Code here
{/exp:channel:entries}
{if:else}
{exp:channel:entries channel="site"}
// Code here
{/exp:channel:entries}
{/if}
It is recommended to refactor the template as follows:
{if segment_2 == 'test'}
{exp:channel:entries channel="test"}
// Code here
{/exp:channel:entries}
{/if}
{if segment_2 == 'another'}
{exp:channel:entries channel="another"}
// Code here
{/exp:channel:entries}
{/if}
{if segment_2 != 'test' && segment_2 != 'another'}
{exp:channel:entries channel="site"}
// Code here
{/exp:channel:entries}
{/if}
disable=’’ Parameter
The disable=”” parameter is extremely important and should be used on every channel:entries tag where applicable. If final rendering includes no member_data, disable it. The same goes for the other options. In addition to rendering the variables useless, it will reduce the number of MySQL tables needed to be JOINed on to provide the data the tag requires. This will help to cut down on the MySQL server CPU Load and Memory Usage.
Using Embeds Wisely
Embedded templates are a wonderful way to abstract your templates and keep your development DRY. I have noticed that they tend to be overused. Each embed requires a query and an iteration through the ExpressionEngine Template Parser, which leads to overhead. So what’s important here is to use the right tool for the right job. In addition to embedded templates, you have Global Variables and Snippets in your arsenal. Fresh Variables are distributed as a first-party addon for ExpressionEngine 1.x, and were incorporated into ExpressionEngine 2 Core as Snippets. Snippets can contain any number of ExpressionEngine tags and are added to the template early in the parse order, along with Global Variables.
As a general rule, if you have an embed that contains static HTML such as Google Analytics Code, or pieces of your HTML header, drop them in a global variable. If you have a sidebar on every page that doesn’t change depending on the URI, drop that code in a snippet. If you have a chunk of code that relies on dynamic, per-template variables or need a small bit of PHP executed in your template, an embed is a fantastic choice.
For instance, we can slice up a template like:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<link rel="stylesheet" href="{stylesheet=master/css}"
type="text/css" charset="utf-8">
<title>
{if segment_2 == 'article'}
{exp:channel:entries
disable="pagination|member_data|categories"}{title} | {/exp:channel:entries}
{/if}
My Awesome Blog
</title>
</head>
<body>
{exp:channel:entries channel="news" disable="member_data" limit="10"}
<article>
<h1>{title}</h1>
{body}
<p><a href="{url_title_path=news/article}" title="{title}">Read More…</a></p>
</article>
{/exp:channel:entries}
<aside>
<h3>Categories</h3>
{exp:channel:categories channel="news" style="linear"}
<a href="{path='news/archives'}">{category_name}</a>
{/exp:channel:categories}
</aside>
// Google analytics and Mint tracking code here
</body>
</html>
Into the following:
{html_top} {!-- Global Variable --}
{html_meta} {!-- Snippet containing title, link, meta tags --}
</head>
<body>
{exp:channel:entries channel="news" disable="member_data" limit="10"}
<article>
<h1>{title}</h1>
{body}
<p><a href="{url_title_path=news/article}" title="{title}">Read More…</a></p>
</article>
{/exp:channel:entries}
{sidebar}
{footer} {!-- Global Variable with analytics code and ending body/html tags --}
And there we have a beautiful template with all the DRY bits we want and we’re using the right tools for the right job!
Relationships
While a very important feature, one to many relationships can be taxing, regardless of the platform. ExpressionEngine works around this by caching the result of relationships in the exp_relationships table. As the size of the table grows, a performance hit can occur as relationships for that entry are recompiled.
PHP in templates
On a high traffic website, one should be very cognizant of what PHP in the templates is doing. Looking through Template Debugging, you will see the start/stop time of PHP processing on Input and Output. If you notice a disproportionately high amount of time for the processing, you can troubleshoot by inserting Template Debugging markers into your code.
For instance:
<?php
$this->EE->TMPL->log_item('starting the query to grab channel title info');
$qry = $this->EE->db->select('channel_id, url_title')
->get('channel_titles');
$this->EE->TMPL->log_item('finished up the query');
if ($qry->num_rows() > 0)
{
$this->EE->TMPL->log_item('starting the loop');
foreach ($query->result() as $row)
{
// Do stuff
}
$this->EE->TMPL->log_item('Finished the loop');
}
?>
Adding the $this->EE->TMPL->log_item() calls, you will be provided with more granular information on the processing time of the PHP. We do, however, recommend PHP be moved to Plugins. This prevents accidental removal of code a front end designer may not be familiar with, which could result in PHP errors on the site. Additionally, you will have reusable DRY code and should see a performance boost since the PHP in your template will not be processed through the PHP eval() function. Nonetheless, take advantage of $this->EE->TMPL->log_item() while you’re building these plugins!
Addons
One of my favorite things about ExpressionEngine is the add-on culture. I do, however want to caution you. Be very judicious in exactly when you decide to use an add-on. An add-on, extra module, extension, etc can add overhead to every page load. If you’re not using an extension, module or fieldtype, uninstall it. Keep your add-ons up to date. If you notice a performance issue an add-on introduces to your site, work with the add-on developer to get it fixed. If you have PHP chops, try troubleshooting it yourself, and send a patch back to the original author.
When choosing add-ons, I caution you to not jump to install it just because you see a link on Twitter, BitBucket or GitHub. Look at the developers history. Do they work well with website developers to ensure bugs get fixed? If you know PHP, do a code review of the add-on to ensure it lives up to your high standards. I’m thankful we haven’t seen it in our community, but an add-on could be released with malicious code which could compromise your application.
I’m by no means saying there isn’t a wealth of amazing developers producing extremely high quality add-ons for ExpressionEngine out there. Moral of the story is, do your homework first.
Server Tuning
ExpressionEngine can run on a wide variety of hosting configurations and setups. We strive to make it compatible in the worst and the best of hosting situations. If you are self-hosting, there are a few things to remember.
As you scale, jumping to a single server to handle all of your web and database traffic is usually the wrong move. If you have MySQL and your web server of choice properly configured, they will be constantly competing for resources when they are on the same box. The key here is resource distribution, and your first step should always be to move to using a separate database server.
MySQL
ExpressionEngine is setup to run using MyISAM. In general, it is not recommended to mix MyISAM and InnoDB on the same database, as they would be sharing resources for cache and buffers.
Web Servers
Apache is our web server of choice, although a lot can be said for running Nginx with PHP-FPM or PHP-FastCGI if you are running a small VPS where resources are limited.
As your website grows, the first step to scaling should be moving your database to a stand-alone database server. From there, load-balancing front end web servers is next. We highly discourage separate installations of ExpressionEngine per web server, so NFS (Network File Storage) should be used to house a single installation that can be seen by each of your web servers.
If you find yourself out growing that, the next step is to move to a dedicated “static assets” server, and I highly recommend Nginx to do this job. Nginx is small, sleek and serves static files lightning fast. This will free up resources on the Apache servers to do what it does best, serve your PHP. Again, using central NFS storage makes this transition extremely easy, as you simply mount a drive, configure Nginx, change a few settings in EE to serve from a new subdomain, and you’re off to the races.
At this point, if your visitors are having slow download times getting static resources, I recommend moving to a global Content Delivery Network. There are two kinds. One that reverse-proxies to your file server and then caches the files on their distributed nodes, or one where you need to upload the files to a third-party service such as Amazon’s Cloud Front, or Rackspace Cloud Files. For me personally, I find the former easier to manage.
Static Resources
A much lesser concern, however a perfectly valid one is ensuring your templates are valid HTML, and you are not making requests to resources that do not exist (404s). Moving Javascript loading to just above the closing body tag is recommended by Google Page Speed and Y-Slow. Sending far-future expires headers on all static assets is a good idea as well. It is also recommend to use CSS Sprites and combine & compress CSS and Javascript when possible.
The reason I say this is a lesser concern is most performance issues start with high database usage. When rendered templates have a low number of queries, and CPU/Memory Resource usage on the DB server is at an acceptable level, this is the next step to troubleshoot.
Care should be taken to ensure it will be easy to offload your requests to a Content Delivery Network (CDN) if needed. So I recommend keeping all static resources in a single directory eg: “/static/” or “/media/”
Load Balanced Web Servers and Caching
Caching in ExpressionEngine utilizes flat files that are written to a cache directory. This works very well when running on a singular machine, however when moving to running ExpressionEngine in a load balanced setup with centralized NFS storage, we do discourage overly aggressive tag and database caching. Your database server should be adequately tuned to be serving as many query results from memory as possible. This operation is faster than reading a cache file and transferring the contents of said file over the network to the web servers so PHP can manipulate it as needed.
Additionally, be judicious with how often you use tag caching. When the cache clears, you can run into performance issues when the tag caching directory needs to be cleared. There are extremely smart and elegant ways to use tag caching, such as caching a REST call that appears on a single page, or adding quick caching on a particularly heavy single usage plugin you have written. However, I advise against using it many times on every page.
Benchmarking
If you are developing locally, while paying attention to your output profiler and Template Debugging, it is recommended to benchmark as you make changes to your templates. The best ways to do this are using Siege or Apache Benchmark.
Siege is easily available to Mac users by installing it with Homebrew and “ab” comes standard on every Mac with the built in Apache installation.
Example usage of Siege:
siege -c 3 -b -t 10s http://example.com/index.php/test
Lifting the server siege... done.
Transactions: 540 hits
Availability: 100.00 %
Elapsed time: 9.60 secs
Data transferred: 0.04 MB
Response time: 0.05 secs
Transaction rate: 56.25 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 2.98
Successful transactions: 540
Failed transactions: 0
Longest transaction: 0.12
Shortest transaction: 0.04
So you can see a total of 540 hits in a little less than 10 seconds. Additionally, you can see we are doing 56.25 transactions a second.
Example Usage of Apache Bench:
> ab -n 1000 -c 3 http://example.com/index.php/test
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking example.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: nginx/0.8.53
Server Hostname: example.com
Server Port: 80
Document Path: /index.php/test/
Document Length: 117 bytes
Concurrency Level: 3
Time taken for tests: 16.253 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 634000 bytes
HTML transferred: 117000 bytes
Requests per second: 61.53 [#/sec] (mean)
Time per request: 48.760 [ms] (mean)
Time per request: 16.253 [ms] (mean, across all concurrent requests)
Transfer rate: 38.09 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 39 49 3.7 49 66
Waiting: 39 49 3.7 48 66
Total: 39 49 3.7 49 66
Percentage of the requests served within a certain time (ms)
50% 49
66% 50
75% 51
80% 52
90% 53
95% 55
98% 57
99% 58
100% 66 (longest request)
Given this output, we can see 1,000 requests took approximately 16 seconds and approximately 51.63 requests/second. As you make changes, continue to benchmark, you will easily be able to see if template code is causing performance issues if there is a fluctuation in those numbers.
What we (EllisLab) are doing to make EE better
As a member of the EllisLab Dev Team, I can say we are constantly striving to tweak and optimize ExpressionEngine so it can take even more. As our fearless leader Leslie Camacho has stated, we’ve broken the dev team into two separate teams. A feature team, and a core team.
Enterprise Services
EllisLab is rolling out Enterprise Services to help developers of large, high-traffic websites with problems such as these. If you want to take advantage of this, please contact .(JavaScript must be enabled to view this email address)
Conclusion
I hope this gives you a bit of insight on what I do when troubleshooting ExpressionEngine-based websites having performance issues. If you’re having these problems and they are because of a sudden traffic influx, congrats. You’re having issues the vast majority of developers can only dream about.
So do it. Roll up those sleeves, put your thinking cap on, and fix it up!
