I’m hoping to start a discussion on caching/performance best practises for EE.  If some good comes out of it I’ll attempt to right up the best bits into a tutorial or case study on Jambor-ee.

I’ve seen more threads started on performance issues lately (maybe I’m just hanging around the forums more?) and I think as we all start building bigger sites utilising more and more of the features we gratefully receive from pMachine these issues will keep croping up.

The caching feature is one area of EE that I’ve never been 100% sure of.  I mean, I use it but I’m never sure if what I’m doing is optimal for the site in question i.e. if the act of EE regularly clearing the cache I create is actually effecting performace more than caching pages in the first place.

I’ve read loads of previous posts but none I’ve found can give a definitve answer - I know there probably isn’t a definitive answer because all sites are different but best practise or some guidelines for certain types of sites are the next best thing.  If I’ve missed any useful wiki/howto/forum posts I’d love you to post them.

So, a couple of questions to get the ball rolling…

1. Say you have a page where the main content cannot be cached because of it’s dynamic nature.  If I split the page up using embeds and cached all subtemplates that I could, would the negative perfomance from using embeds be out-weighed by the positive performance gained from the caching?

2. Say I have a sidebar that displays the titles of the 10 latest blog entries.  Would I be better using the exp:weblog:entries tag along with both the disable parameter and the “Cache Dynamic Weblog Queries?” option, or simply write a custom query to pull only the basic data (title, url_title, etc) that I need.  Basically, I could write a much lighter query using the query module but does the disable parameter and the fact I can cache weblog:entries cancel that performance gain out anyway?

If anyone’s willing to share info on their own setup along with site details that’d be useful too.

Yea, this is one of the areas where I’m still not sure either- and I think it’s because the general answer boils down to ‘it depends’.  I used to think I totally ‘got it’- but then Nevin laid out some of the complexities involved on the server side of things, noting that PMH recommends turning query caching off, as the servers are already handling query caching and it just adds overhead to duplicate the effort.  For your number one, if I wasn’t on pmh hosting, I’d probably not embed simply to be able to cache- but you can still cache tags- including the query tag.  That would probably be the option I’d take.

I generally will cache things (i.e., template caching) that are consistent- header, footer, nav if applicable.  And I cache the index page if it has a lot of queries on it- which it generally does.  Single entry and archive pages, I don’t.  Generally.  I end up debating the ‘weblog tag vs query module’ a lot as well, because you can write a leaner query most of the time.  But then I figure if someone else ends up having to dink with the templates, it’s better to use the weblog tag- and with disabling it really shouldn’t be that big a deal.  But it’s one of those things I’d reconsider if I was trying to get things as lean as possible.

Hm- this was kind of a ‘non-answer’, but I figured I’d bump the thread with it .

I hadn’t seen that thread before Robin - it’s an interesting read, thanks.

And thanks to everyone else for your input, it’s been fascinating

smallbeer - 09 January 2007 07:01 AM

And thanks to everyone else for your input, it’s been fascinating

Hm, sarcasm, right?

On the face of it, I may not be the right person to jump into this thread since I’m only on my fifth day of the 30-day hosted demo. However, I’ve spent 30 years in the IT business, including a few years as a mainframe performance improvement specialist, so I’m not new to the performance issues that the EE developers are facing.

To be honest I’ve already made up my mind. I just love this product - at my age you learn to trust your instincts - and I want to start building my first production site now. But the performance fanatic inside me complains about the reports of performance problems in these forums. Furthermore, it reminds me that sometimes developers need to stop adding features and spend a release or two just improving what they did so far. Now I certainly don’t know if this is a problem in this case since I’ve no record of the EE release history or development plan, I’m just too familiar with how developers think in general. Function is fun, performance is boring. And much more difficult.

Feel free to visit my first demo site at http://demo.pmachine.com/r32z0/index.php. (When the demo is dismantled in 25 days there should hopefully be a production version at itinthemiddleages.com.)

It’s a very basic blogging template with some static pages in the About section. At the bottom of the left-hand column you can see the number of database queries (Q) and the script time (S). The site is untweaked in all regards so this is what I got just by developing it without any previous knowledge.

Reading the performance hints in the wiki and user’s guide, I already know of some of the problems the site is having:

Do not use embedded templates.

Now, I just can’t accept that. That really says instead of pre-compiling the templates, the developers use them as-is for every request, which means lots of SQL queries. That’s not a user tip to me, it’s a feature request.

Don’t needlessly use the database. For example, don’t use a separate blog to run a link list.

Well, I did a search for “static pages” and the recommendation was indeed to use a separate weblog for them. Once again I see a feature request - make it easy to create static pages without any performance penalties.

Use the “tag disabling” feature, which lets you turn off things in your weblog tag that you may not be using.

Well, you know what I’m thinking. Feature request. Why was EE designed to fetch things until they are needed? Mankind can still build flight reservation systems, why not use some of that design experience in the EE development process?

Don’t run sessions. Use cookies only.

Ouch - sessions are so neat, and cookies are so clumsy. But I can bend on this one.

Enable Dynamic Weblog Query Caching

Sounds great until you find the following little note: Enable this feature only if you do not use the “future entries” or “expiring entries” feature. Well, I certainly intend to, I think that’s one of the many great strengths in the product. And I keep thinking that 1) it would be easy to cache whenever possible, and not to do it when it isn’t; and 2) it should never be up to the user when and what to cache - the system has all the information it needs for the right caching decision.

Don’t use plugins which require database access.

I can use a text editor and create some really fast web sites. But I’m here for the plugins, and all the other fancy stuff that makes EE a product worth loving!

Don’t get me wrong. I’m really impressed with the overall quality of EE, and I don’t want to criticize the people that wrote the code or did their best trying to document it. I will learn this, follow the rules and my sites will run fine. Most of them are very small anyway.

I’m just noting that there are quite a few reports on performance problems in the Troubleshooting section of this forum, but almost nothing of it seems to find its way into the Feature requests section. Instead, there is a growing section of the documentation saying “don’t exploit the product’s full potential if you want to build sites that work”, and now you’re trying to add your two cents of documenting design deficiencies instead of correcting them.

Heck, I’m in the beginning of a love affair here, and you’re telling me she currently can’t handle much more than remote admiration?

Smallbeer,

Great idea for discussion.  My personal thoughts on your point number 2 are that a query would be the better option.  The reasoning behind that conclusion is simple - the entries tag, although cached, needs to run several queries to grab the cached data anyway.

On to one of my own practices here’s a technique I employ in nearly every project i do.  Say you have a template, looking something like:

You will notice that, although you have only asked for either of those entries tags, both pairs will get parsed out by EE’s templating system. Not good, if either of those circumstances are alot more involved than just a single pair!

Thanks to PHP on input, we can call the $IN variable, and manipulate what we send to the templating system thus removing that extra processing who’s result never gets seen! example:

I never knew that, and I use similar code in various places on most projects too.  I take it both pairs only get parsed if you’re using advanced conditionals?

Glad the discussion is picking up.  Well done for spotting the sarcasm by the way Magnus, did the wink give it away?

AFAIK it applies to any conditional, ee will look at the content and run through it anyway.

LOL- no really, the sarcasm was very subtle.

I think….  There’s a trade-off between designing a site to be as lean as possible and keeping things user friendly.  Like on my own site- I’ll generally use the_butcher’s approach to complex conditionals, because it will be leaner.  (And you’re right, it’s only for the complex conditionals that it all gets parsed).  But- for newbie advice on the forum?  I use the EE conditionals.  I don’t want to try walking them through the php- and it scares them.  Same thing with the decision to use a query vs. a weblog tag- I might use a query for myself, because most of the time you can write a leaner one than the weblog tag will reproduce- you can hard code things, not bother with the extra queries to translate weblog names into weblog ids, or pull up the custom fields available and their shortnames…. etc.  BUT- I sure don’t do that for forum advice- again, you’ll just end up confusing folks.

And some of the advice on making things as lean as possible?  Don’t use embeds?  Well- true, it is leaner, but most designers would chalk them up as one of the most useful things to have.  I ALWAYS use them for headers, footers and navigation.  And I like my stats module- thank you!  And my sites runs fine.  Though granted- they aren’t getting slashdotted or anything.

So in general, I think go with the best REASONABLE practices- use the disable parameter.  Avoid having 50 billion weblog tags in a long list of complex conditionals.  Watch out for nesting queries, which folks do all the time.  Sometimes for good reason, but at least know what you’re doing when you do it.  Make sure you have a good host that’s capable with dynamic sites.  Then if you’re having troubles, start the hard core optimizations- or consider the need for more horsepower.  The flexibility and user-friendliness of the weblog tag does come at a bit of a performance price- as do some other features.  But for most folks, it’s going to be a non-issue.  When it becomes an issue is when it’s time to get real creative.

(I’ve got a site- magazine style- each ‘section’ is a weblog.  They want all of the articles for an issue displayed in their section on the main page- and the sections in a particular order.  Took maybe 12-15 weblog tags to do it- which made me cringe.  But it does run fine.  Of course, it bugged me enough I just got around to writing a plugin that will allow me to use one tag and order the sections according to a parameter- because it was seriously ugly and bugging the crap out of me.  But the client never noticed the difference.)

And welcome aboard, Magnus!  Hopefully the dev crew will chime in on the topic.  You guys are raising interesting issues.

Magnus I felt the need to hop in and comment on the few things that I can in your post, as it’s well written and deserves a response.  Some of the things that you see people using and recommending in the forums are not necessarily going to be best practice.  Even at times when a staff member might recommend a specific solution for a user, it may be tailored to that user’s experience level.  Because for the most part EE stays out of your way when creating your pages (templates), there are literally an unlimited number of ways to approach site building.  That Knowledge-Blog article you reference is written primarily for people with serious template issues.  Nesting tags, dozens and dozens of embeds, poorly written 3rd party plugins or self-written PHP, and some very…weird…organizational concepts.  When you are using weblog entries with custom fields used to output parameters to another weblog entries tag, it’s obvious that firm intervention is needed. I mean that with love of course! 

We do our best to keep things as light as possible, and look for ways that we can continually optimize the application in the future.  But one of the greatest features of EE, the flexibility, can sometimes cause an inexperienced developer to get themselves in trouble with regard to resource usage.  So until you become very familiar with EE, start with those restrictive recommendations, and work from there as a starting point.  We fully expect people to take advantage of every nicety of our sports car, but new drivers need to wear safety equipment and maybe even keep the governor installed to limit their speed until they get used to the handling.

So, all that said, do you really find a .3 seconds rendering time to be an issue?

To Magnus, mostly:

Also, in regards to performance issues - please remember that while a few of us come here many times a day just to help and chat, most people will come to forums when something is wrong.  That spans all forums, tech and non-tech.  I sought out a cat forum when I was having behavioral issues with my kitten, for instance. Never went back.

Also, most of the performance issues can be traced to a few issues that we can’t do anything about.  1) People hosting their own websites without the requisite knowledge to tune or properly run a server.  2) Over-run shared hosting servers that simply aren’t up to it, usually also run by administrators that don’t have any idea what they’re doing,  and 3) Getting pounded by spiders/bots and not using the blacklist and throttling abilities that were put in to abate that problem.

Oh, and 4) overzealous templates without realizing that 500 conditionals does take time to process.  And 5) overuse of .htaccess, remember - .htaccess is utilized for every single thing a webpage does - grab a graphic? server checks .htaccess.  You can ruin any site by doing crazy things in that file.

As far as the wiki article on optimization - I really should update.  Do remember the wiki is a community resource, updated *by* the community.  Some of those options (not using embeds, etc) are last ditch options, and not generally needed.  After all, things like that are basic features of EE that we suggest to use day in and day out.

If you go looking for threads on performance issues - yes they’re there, but they come up once every few months, usually, though we occasionally have “spurts” of them.  But if you read the forums regularly, you’ll find that they come in relatively rarely.

I wasn’t meaning to imply that there were performance issues with EE btw.  What’s coming out of the thread is what I was hoping for though, that if you’re happy to lose some of the flexibility EE offers you there are more options for “experienced” EEers.

I’m not sure where the ‘loss of flexibility’ comes into play here, smallbeer.  The flexibility does not change with experience, only the potential performance of such flexing.

Derek Jones - 10 January 2007 09:14 AM

So, all that said, do you really find a .3 seconds rendering time to be an issue?

Ha ha, certainly not. But perhaps the ~45 SQL queries that were required, when the Wiki states that a default installation of EE can be brought down to 4 queries. I don’t like to do embarrassing things, and I prefer to have a reasonably good design before I throw hardware at a performance problem.

But really, I’m not complaining about poor performance here, my sites are small and will probably fit comfortably in just about any hosting environment. It’s just that I’m right in the middle of buying EE, and my hosting operator wants to run it on a VPS, and then I see John Fuller shaking his head at the notion, and then I read the recent performance-related posts and the corresponding responses from pMachine, and suddenly it’s just deja vu.

When I was working for IBM, I wanted to be a programmer. “OK”, they said, “as long as you understand that programmers mostly do other things than write programs. They have endless arguments about requirements and then they need to test and document the requirements. Then they have to design, and what’s even worse, they will sometimes have to redesign. If something is difficult, they have to build various models or prototypes and then they need to test and document the models and prototypes. Did I mention that they of course have to document the final design in detail? Anyway, then they spend a few hours writing the actual code, after which they must test and document it. Once the program is released to its users, programmers spend lots of time providing training and support. And when they have a fix for a problem, they have to test and document the fix. So, do you still want to be a programmer?”

And you know, I said no, I just had to. I just wanted to be a programmer, not all those other things. But after that, my relationship to developers has always been a bit tense, as in “why do they get to have all the fun, and not me?”.

Derek Jones - 10 January 2007 09:38 AM

I’m not sure where the ‘loss of flexibility’ comes into play here, smallbeer.  The flexibility does not change with experience, only the potential performance of such flexing.

By flexibility I meant for example using php conditionals instead of EEs built in ones - it’s now more susceptable than it was…less flexible?

Fair do’s though, maybe it’s the wrong phrase.

Magnus, you bring up issues of concern to me. I am in a quandry at the moment using EE, especially as it’s the only platform I use. People frequently say it can do anything content-related and I believe it. And I go ahead and start building what seems to me and the client to be the best way to present data to the user in order to accomplish the site’s goals. This tends to require much interrelating of data from different weblogs, which requires much embedding of templates, which begins affecting performance to the point of unusabilty. In the forums here I’ve effectively been told that my use of the system with extensive embedding is abnormal. So at this point I’m unsure where to turn. Maybe to CodeIgniter.

Adam,

I am new here and I have no real experience of EE in any way. I don’t seriously think there’s anything basically wrong with the design of ExpressionEngine.

In the design of a user-friendly tool like EE you never try to cover 100% of the market’s requirements. I think pMachine should be proud to have a product with the wide range of possible uses like EE, and if you look around you will see them getting much credit for this, both here and in many other places. Reading such positive comments is what brought me here, and many others.

If you have special requirements, it may be that you will find a hard time finding any general-purpose tool to help you. Many high-performance web applications are still built using traditional programming languages with raw database access.

I’m not speaking of web applications, but of non-interactive sites that bring together on a single template inter-related data from a variety of weblogs. I’m amplifying your own notion:

Magnus Wester - 10 January 2007 03:34 AM

there is a growing section of the documentation saying “don’t exploit the product’s full potential if you want to build sites that work”

What I want is to feel fearless about bringing aspects of all my site’s content to bear on one screen view, without having to make templates that consist entirely of the query tag. Certainly, there are steps being taken towards this, such as related and reverse related entries, but these aren’t yet nestable. Perhaps they will be soon and I will once again be a happy camper.

I’ve had some big performance issues with ‘comments’-module, ‘category’-tag and ‘calendar’-tags. For building these sets of a few entries the script searches the whole database. ie. I have 20.000 comments and then it became horrible.

My tip for lists like last 10... most commented... last month... etc. (the stuff you put in a bar): use per list a special embed-template and cache it.  That’s all.

When I found this solution my sites served 300% quicker!

Ralph those tags have built in default limits to prevent unintentional query creep, so they do not grab results from the entire database, but serving them from cached embedded templates is a fine solution.