Thread

Earlier tonight my EE based website was temporarily deactivated by my client’s hosting company (Lunarpages) because they said the site had exceeded it’s server usage levels.

I’m using a mixture of template and tag caching, along with query disabling, to help make the pages run as efficiently as possible. Unfortunately, even with these performance features in use I’m still running into problems.

I’m hoping someone might be able to take a quick look at the code on the page that seems to be the biggest resource hog to see if I’m simply building my pages in an ill-conceived manner. 

Here’s a specific snippet of code I’m curious about:

{exp:channel:entries dynamic="yes" channel="programs_workshop" dynamic_parameters="category|status" category="1|2|3|4|5|6|7|8|9|10|11|12|13" disable="pagination|member_data" status="open|Full|Waitlist|Postponed|In_Session|Not_Yet_Available" sort="asc" orderby="start_date"}
    {if no_results}
        {workshop_no_results}
    {/if}    
    {if url_title!="coming_soon"}
    <div class="workshop_holder">
        {workshop_description_holder}
        {workshop_info_holder}
    </div>
    {if:elseif url_title=="coming_soon"}
    <div class="workshop_holder">
        {workshop_description_holder}
        <div class="clear"></div>
        </div>
        <div class="workshop_bottom"></div>
    </div>
    {/if}
{/exp:channel:entries}

Also, attached is the code for the entire page.

As I mentioned, the site is currently down. Once the site is back online (which I hope happens soon), you can view the page in question here:
http://www.826chi.org/programs/workshops/

Any insights would be greatly appreciated.

Thanks so much.

bburwell, here’s a terrific article by Greg Aker, EE hosting and code guru, which rounds up a number of performance practices.

In a quick look at your code fragment, two things jump out at least, which are early points in his list:

- You are using an ‘advanced’ conditional—i.e. if:else—to select two alternate responses. But the way these work, due to parse order, EE actually completely prepares both possible results, which costs you for both. The simplest and effective answer is to use instead ‘simple’ if/endif along with appropriate conditions, once for each branch.

- You are not using the disable= parameter on your channel:entries tag. This can be very important because it avoids database joins which will otherwise be made to prepare lots of data you won’t probably use, for each entry.

Greg runs you through using EE’s profiler to locate especially long operations, which can help you home in on what’s important, and then get at least a proportional measure of what you’ve achieved, needing to be interpreted for other load on the server of course.

The one thing he doesn’t hit this time is specialist plugins which actually serve flat files based on your EE pages. I don’t know what this site you have is or what it does, but if it sometimes gets hit very hard (gets mentioned elsewhere and gets a day of hits) these could possibly be useful to stay within your cpu limits.

I wouldn’t go there until you’ve seen what you can accomplish by cleaning up basic efficiency of your templates, both server and database use, especially as you have the if/endif choices where activity can’t be in static files.

However, if it turns out appropriate, Solspace has items mentioned here which have become available for EE2. I’d probably get some consulting up front if you think you might be tempted towards measures like that.

Last point. Again if this is a site which gets hit hard, are you sure what the hosters are excited about? Is it load on servers, or is it Internet bandwidth used, or both? That points out another area to look at: size of your images or any other page-downloaded items. Images can often be much smaller (factor of 10) than the ones people sometimes use, without visible quality loss, if this is a problem. You’d use the ‘export for web’ abilities of a Photoshop or Fireworks, for example, to do the optimization.

Well, lots of avenues. EE should be quite reasonably efficient, given all normal correctives are in place. I’d read Greg’s article, and see what you can do from there. Should be interesting, and will wish you good fortune in it.

Regards,
Clive

Thanks for the assist, Clive.

I’d have to echo Clive here in that there are things you can do to improve .. using single ifs, using the disable= parameter, etc. Take a look at what Greg posted and what Clive mentions. See if that helps.

Thanks, Sue.  bburwell, wanted to note that I’d missed in reading that long channel:entries tag last night that you actually did use disable=. 

Are you able though to use it more, though? If you don’t need custom fields or custom channel fields in this particular template display, you can add those via custom_fields and category_fields respectively, and save more database joins.

Once again, I’d change that if:else for separate if statements right away. It looks awkward from a programmer’s old perspective, but is needed in EE so that you save evaluating every possibility on each page load.

Beyond that, and after having a squint at the channel:entries code again, I’m reminded of some long knowledge that everyone who has done optimization agrees with. 

It is to always measure, rather than presuppose. That’s because the bottlenecks are so often not where you expected, or there are more of them than you suspected.

Then Greg’s suggestion to use the profiler would be your next stop, given again that CPU use is a big point in the hoster’s complaint.

Similarly, you can use browser developer tools and/or the excellent Google Page Speed Online to look at page elements for bandwidth use.

Regards,
Clive

Hello.

Thank you both for the thorough, timely and thoughtful feedback.

After fine tuning some of the caching and performance features, I pulled some data from EE’s output profiler that I’m hoping you might be able to quickly review for me.

For the most resource/database intensive page on the site (http://www.826chi.org/programs/workshops/), below are some output numbers from before and after caching (which I recorded after doing a quick clear of the system cache):

Before
Total Queries: 50
Range for Query Times: 0.0100–0.0001 (with most numbers being closer to the smaller side)
Total Execution Time: 2.5 sec
Memory Usage: 8,037,068 bytes

After
Total Queries: 33
Range for Query Times: 0.0100–0.0001 (with most numbers being closer to the smaller side)
Total Execution Time: 1 second
Memory Usage: 7,850,000

Based on what I read in Greg’s article and comparing against the output data from the EE system pages themselves, they look like they fall within the realm of what’s reasonable regarding server resource usage. But this isn’t an area I have much knowledge with so any confirmation or opinion to the contrary would be helpful and much appreciated.

Based on some online reviews of my client’s hosting company I’m led to believe that they have exceptionally low server resource usage limits. Knowing that, my goal is to confirm that I’m coding my site in a reasonable and logical manner when it comes to site performance, and potentially determine if I need to consider recommending that my client move to a different hosting provider.

Again, thanks so much for your assistance.

best,
Brett

Brett, as well as your post here of results, I’ve had a pretty good look around your site, now that it’s back up. Here are some thoughts, besides that the site’s purpose is also interesting:

- My general feeling is that your numbers reported, reflected also in the measured site details performance using browser developer tools, looks quite reasonable. Even the graphics are under control.

- you are of course using a lot of feature functionality, such as the web fonts, and they cost something, especially for a client’s first page load. For the page of interest, it came to just about a megabyte to load the entire page, which is high, and not everything was freshly loaded - the font apparatus had hidden something away in Flash’s cache.

- nontheless, I think the site is working normally. Your improvement figures with following through Greg’s notes tell a story: you got a lot of improvement, well done.

- this particular page, for workshops, is still very heavy, if not unusably so unless on a slow connection. Because it’s long text content is generated by PHP as well as wire reloaded every time someone accesses the page, it is quite likely contributing to your hoster’s concerns.

- I’m thinking about whether such a long list of workshops is really in the site client’s interest? Will they really want to page through all this?

  == To cut down this load, you could add a layer of hierarchy; break the workshops page into some kind of tags or categories, as you did the programs page above. This will cut each category page and the overview into performance loads quite like your other pages.

  == It occurs to me that another path to reducing the load could be to take a look at what the workshops page items are - quite succinct, similarly blocked-out information in each. They look then like they would be quite a good fit for the Pixel&Tonic; Matrix add-on, which is kind of wildly popular with EE developers for catalogs and the like. Matrix looks that it should be more efficient at delivering your workshops elements, because it does so out of its own table. It also is probably an attractive solution on the data entry side.

  == I had further thoughts about design possibilities, such as making a more direct interaction panel framework for the workshops, but let’s leave that as a site attraction and usability factor. I don’t think it would significantly alter performance from a more straightforward second hierarchy of tags or categories.

To recapitulate, then, your graphics look ok, and there’s a definite but reasonable price paid for the web typefonts. All of your other pages show PHP processing time of 0.5 to 0.9 seconds, each in proportion each time to the size of content rendered. Those times are entirely reasonable for ordinary server arrangements.

This problem page, on the other hand, takes 2.7 seconds or so to render in PHP, which is a lot—and again it looks entirely proportional to the large group of channel entries you’re processing there. This is where the suggestions to break it up by tags or categories feel indicated. If you also choose going to Matrix, you may be able to pick up another basic efficiency gain, though you will have to try it to see if that’s true. if so, it may help ease the door to those future design possibilities.

Hope that’s fair, Brett. I don’t have anything to do with Pixel&Tonic;, of course.

As well, others may well have more to say which will be helpful, especially if I’ve missed something, easily a possibility.

Good fortune on the site, and regards,
Clive

Hi Clive.

You should win some sort of EE award for your forum support. This is an incredible followup. Thank you.

I should have clarified with my original post, but the issue I’m bumping up against is 100% related to CPU and memory usage, and not bandwidth. I realize I’m taking a page load hit for the web fonts and other visual items on the site, but in general I’m very pleased with the page load speed (even on slower connections). Sorry if that added extra steps in whatever analysis you did.

Some followups to your notes:

1) Good to know that you feel the site is generally running normally. I’m still learning the best ways to build an EE based site, but it’s helpful to know that I’m generally on the right track. So thanks for confirming that.

2) Splitting the Workshops page into smaller chunks is a good thought. The client initially requested that they all be displayed when you first come to the page, but it sounds like it will be important to reconsider that.

3) I’ve used Matrix before, but wasn’t familiar with it when I fist built this site. Good suggestion. I poked around Pixel and Tonic’s site (along with some general quick Google searches), and didn’t find anything about the performance benefits it might bring. None the less, it might make the administrative content management easier, so I appreciate the thought.

4) What I still don’t understand is why the hosting company took the extreme step of manually disabling the site due to excessive CPU/memory usage. Based on the numbers I was seeing in addition to what you reported back, I realize that the “problem” page is more resource intensive, but not exponentially so. The host reported that the site was using 9% of the total server CPU resources. Even with a temporary spike in traffic to that page, it feels like that shouldn’t be possible in a “good” hosting environment (which I realize is a relative term).

Again, I can’t thank you enough for such a detailed response. Your insights and perspective are very much appreciated.

all the best,
Brett

Major thanks to Clive, yet again, for being so attentive to our community.

Brett, have you had a chance to check with your host to see what kind of CPU usage they’re seeing after you optimized your templates? The before and after times on your queries alone looks like it should be a major gain. Let us know what your host is reporting now.

Kevin

You’re welcome, fellows.

Brett, I mulled the Matrix performance gain part of the equation yesterday while off on something else, as I knew there was something I wanted to remember about that.

The real gain had come over prior versions of Matrix, because they had been encoding their data into standard EE tables. Separating this out into their own table was definitely a performance move in this comparison, as well as reducing complexity. However, I suspect that in your situation the separate table wouldn’t show visible improvement, given that you’ve already used the full set of your disable=; if it definitely shouldn’t be worse.

So I think you’re right in your conclusion, and I knew at the time of writing that my big attention to Matrix was that it had set off some thinking for the possibilities of advanced user interface design—again, not necessary to have it for that, just possibly nice.

As to why the hoster clipped you at 9% of server usage (a single server??), that does feel unusual, and especially to do it without warning you first. Maybe something was misunderstood there. Or, they may have an automated handler with a threshold too low, and without a needed human oversight to ok what it thinks it should do, in communicative and escalating steps.

It is important that shared-hosting operators keep a close watch, as people do put up rogue scripts, often without the slightest idea of what they have done. I remember helping a Swiss host find what was just about halting the big server I was on for minute-long intervals, just not entirely often. In that story, it turned out that someone had moved to South Africa while leaving a calling card site, including a script for emailing new entries in their guest book…which went completely wild each time someone wrote them a note. Kind of a smile after that one.

I agree with Kevin that having a civil conversation with them sounds like an effective path. Then they know who you are, and that it’s not a rogue, and that you’re attentive.

You have the improvement evidence in the figures from your performance hunt, with the knowledge of precisely which URL path is heaviest and why. You might turn the card that you’re considering a further improvement, subject to your client’s agreement. You have for what it’s worth a software community analysis that says you’re doing the right things in managing the situation.

Good fortune on it, Brett, and imagine this will turn out well.

Regards to both,
Clive

Thanks again, Clive.

I’d agree, Brett, that this sounds like a hosting issue right now. You may have simply outgrown your plan, so hopefully something can be worked out.

Hi guys.

Thanks again for the feedback. Sorry for the slow reply, I’ve been away from my desk these past few days.

Clive, thanks for following up regarding possible performance gains from installing Matrix on the Workshops page. It’s good to know that any benefit would likely be negligible.

The host company did indeed confirm that our CPU and MEM usage fell back down to acceptable levels, and even went so far as to say that the site was “loading great.” The problem that I have is that they’ve done nothing to confirm that they won’t again freeze the site without warning when there’s another small spike in traffic.

I’m not sure if I shared the specific details (nor if you care about them!), but here’s the short story. Twice a year the website gets hit with a small rush of visitors when the organization opens up their workshop registration. This spike in traffic usually happens within a tight window of time as visitors are all eager to be “first in line”. Our analytic statistics show that it’s typically around 200-300 people within roughly 30-60 minutes of time.

I realize that a spike in traffic to a page that’s more resource hungry is going to chew up some server resources, but I’d like to think a decent hosting environment could handle 300 visitors to a website that I believe we’ve now confirmed is running reasonably efficiently.

Sadly, I still think changing hosting companies might be our best answer.

I can’t tell you how much I appreciate your continued feedback and support. I initially started using EE because I was completely floored at how flexible and easy it was to use. But the high, high level of quality in the support forums has become a significant reason it’s still my CMS of choice.

I’d consider this a closed issue at this point, but feel free to followup with any last thoughts if you feel so inclined.

Thanks again.
Brett

Awesome, Brett. So glad to hear things are operating at normal again, and thanks for adding those details and the kind words in your follow-up. We strive to offer great support to our community!

As always, feel free to post to the forums again if you come across something you need help with.

Kevin