Thread

How are you making the global headlines list? Do you need more than just the title and the date?

I’ll let Stephen follow-up here because he’s more aware of the specifics, but at the risk of not using proper nomenclature, we use timestamp, URL, title, and category. I believe there’s a bit of filtering that happens to build this list, too. Depending on how many articles from “today” have been published thus far, we may include some of the articles from the previous day, and filter them by their status, just to ensure a full list.

So there’s definitely some logic and calculation happening, but I would be fine if that only happened once every 5 minutes. In fact, with our previous site (which ran on our own, custom CMS), that’s exactly what we did. We generated it once and just included it in every page. Very efficient, and the worst thing that happened was that someone visited the site and *didn’t* see an article that was only 2 minutes old. An acceptable tradeoff for our world.

Nevin, thank you very much for taking the time to provide such details answers.  This has given me a lot to go on, but has also lead me to even more questions.

  Have you fully analyzed the logic in your templates?  3rd party add-on impacts to render times?

Yes.  Many of the embeds / site template rely heavily on advanced conditionals, multiple embeds, and plugins to render our content.  When cached, I can get a page to render in 0.5 seconds.  When not cached, it’s typical to see the weblog:entries part take 2 seconds, comments take anywhere between 2-5 seconds depending on the number of comments, header/footer/navbar embeds take another couple of seconds.

I’m not sure where to start in terms of simplifying templates.  It’s a complex site, and the impression I’ve always gotten about EE is that it’s a blank slate—it may not do everything out of the box, but it has great addon support to make it what you want.  I heavily use imgsizer, primary category, and child categories.  I use MD eexcerpt, html stripper, xml encode to generate my meta description and rss.  I’ve created my own plugin to force only entries to be displayed in eastern time.  I see the log on the template debugging grow and grow, but I’m not sure how to avoid it without sacrificing what I want to display and render. 

But again, it’s night and day between when a template is cached or not.  If cached, rarely does it take over 1 second.  If not, it will take anywhere between 5-20 seconds.

Are you seeing heavy web server resource usage (cpu/ram) per page request?

I see cpu spikes to 15% for threads while they’re processing pages.  We have plenty of free ram.

Are you seeing an impact at the db server level?

Yes, MySQL can easily get backed up, sending the apache thread to 100%, and stalling the site.  Dave is very knowledgeable out MySQL tuning; I’ve done my own research and used many of the common MySQL analysis tools, and our server admins also have gone through the configuration as well.  We’ve eeked about as much performance as we can out of that server (4 2Gz Xeons; 8 GB RAM).  It’s still very typical to ee queries in the slow query log or backing up the processlist while they cull through thousands of rows and copy to tmp tables. 

I’ve taken a look at what queries are the worst culprits.  The worst one was a query to get the next/previous entry in a category.  Although we wanted that feature, we had to greatly reduce it’s use since it was too expensive.  The queries used to generate our category and monthly archives, and the recent headlines widget also appear to be expensive.

Have you considered pulling in outside consulting to assist in analyzing your bottlenecks? ie: Solspace Performance Analysis?

We have.  We thought our questions were better suited for the people who created the product; we wanted the info from the horse’s mouth instead of a 3rd party so to speak.

Further Questions:

It’s very tempting to hack the core files, and bump up that 1000 URI cache limit to something more like 5000 or 10000.  It seems like our webserver could easily handle more cached requests.  Could you explain why that might not be a good idea?  Also, when I look in the cache folder, I see 4 folders:
db_cache: what does this correspond to in the Admin UI… dynamic weblog caching?
page_cache: presumably the template caches
sql_cache: the query caching?
tag_caching: presumably the tag caches

What counts against the 1000 limit?  For example, I just did a count on the number of directories in the page_cache directory (ls -1 page_cache/ | wc -l): 29685.  That doesn’t make sense to me.  Do the files in the sql_cache also count against the 1000?  If not, is there a separate limit for the caches that don’t fall into that 1000 limit?

A good extra source of information if you have a high traffic site would be Handling Extreme Traffic, but really those settings and the disk i/o section can impact even moderate traffic sites.

We do have all but single entry hit tracking turned off.

Unless you actually are seeing a specific need, for a specific template, to be cached, I would highly recommend to not cache unless you have to.

In the above example of my sidebar embed… let’s say I find it beneficial to be cached, I also decide my homepage benefits, but my single entry page doesn’t.  However, since my sidebar is embeded on my single entry pages, does turning off the cache for my single entry template actually reduce the number of files to cache?  If I understand you correctly, that embed has to be cached for every single entry URI whether or not I have that single entry template cached or not.  (In a scenario with a site with 100 entries versus 10000, it seems like the site with 100 entries benefits from the template cache while the 10000 entry site does not).

How are you making the global headlines list? Do you need more than just the title and the date?

Weblog Entries Tag, disabling categories, category_fields, custom_fields, member_data, pagination, trackbacks

Weblog Entries Tag, disabling categories, category_fields, custom_fields, member_data, pagination, trackbacks

Have you considered using the query module and only dealing with the exp_weblog_titles table?

Have you considered using the query module and only dealing with the exp_weblog_titles table?

I did take a look at the query the weblog tag generates, and I do see that I could write a simpler query.  So thanks for pointing out one place to simplify.  But really, I shouldn’t have to be writing custom queries for this.  Why is it still joining exp_weblog_data in when I’ve disabled custom_fields?

I did try disabling both Query Caching and Dynamic Weblog Query Caching, and I cleared the cache, but my sql_cache folder is still getting files.  What else do I need to turn off?

Hi Stephen,

Shameless self promotion here…

We have some recently developed software intended to address a number of the issues described in this thread. Super Search is designed to take the place of weblog:entries on some cases. Template Morsels lets you cleanly cache snippets of site content into the database and using some more tunable refresh rules. Anyway, you’re not alone in the desire to optimize. We’ve got some tools that could help. solspace.com

mk

Thanks to Nevin’s prodding and support on Twitter, I think I will work on building a full document explaining the usage of caching in and with ExpressionEngine, everything from proxy caches to MySQL’s built in query caching.  That will be quite a task and I have a full plate at the moment, so I would not expect it until the beginning of next year.

In the meantime, I also am going to do a bit of shameless self promotion here, as this topic is rather important to me.  I work for Solspace, as its Lead Developer, but prior to that I was the CTO of EllisLab and wrote a significant portion of the ExpressionEngine 1.x branch.  The EllisLab team works on building the software and making it stable.  They provide excellent support, but this is a topic that requires a certain amount of in depth experience and investigation.  ExpressionEngine is complex and powerful and because of this the ability to write poorly performing templates on high traffic sites is extremely easy.  Discovering and solving performance problems in these situations requires time and patience.  There is no all encompassing solution or one, perfect way to do it all.

I suggest you read Solspace’s Performance Guidelines as they are a great way to start solving performance problems and they’re freely available to all.

Another step you should do is turn on Template Logging and look at the processing times between items.  Reading this thread, you are talking about some tags taking over a second to process.  No matter the amount of traffic you are getting that is simply unacceptable for an EE tag to take that long.  Narrow down what tags or steps in those tags are taking the longest.  Create separate templates for those tags and try to solve each problem individually.

Hi Stephen,

Have Nevin and Paul been able to answer your questions satisfactorily?

Cheers

Greg

Were talking to solspace about using template morsels.
And were both very much looking forward to reading the caching paper. Solspaces performance paper was such a huge eyeopener.

Should I report that exp_weblog_data is joined even when disable=custom_fields is on, as a bug?

Does anyone have a good argument against bumping up the internal 1000 uri limit in the core code?

  Is this limit per site or system?  I’m just trying to figure out why I see more than 1000 directories in the page_cache folder. (unless this isn’t a short answer, and will be explained in the paper)

thanks

Have you had a chance to read the submitted literature? If you still have unanswered questions, let us know, and we will try to get them answered for you.

Thanks!

Yep, read through everything.  Those questions are still relevant.

Stephen, Adam’s passed on the details to the dev staff, to see if we can get more answers for you.

Thanks in advance for your patience.

Should I report that exp_weblog_data is joined even when disable=custom_fields is on, as a bug?

I’ll take a look- but IIRC, the disable eliminates another query used to gather the full data.  But I’ll take another look- might could still be improved.

Does anyone have a good argument against bumping up the internal 1000 uri limit in the core code?

Nevin and/or your server admin will be the best to address this- but at some point (the point depending on the site/server setup), you’re just shifting the bottleneck to the Disc I/O.

Is this limit per site or system?  I’m just trying to figure out why I see more than 1000 directories in the page_cache folder. (unless this isn’t a short answer, and will be explained in the paper)

The page cache is system wide.  While there are a lot of acts that clear either page/all caches (search the project for clear_caching to get a feel) the biggie that should be preventing overrunning the max setting is in core.template.php - run_template_engine().  While I didn’t track it down explicitly, I’d imagine it’s adding to cache w/out running that often enough to keep up w/the deletes.  OR there’s a problem clearing caches.  That should be easy enough to test for- clear all caches in ‘Admin- Utilities’ and then check via ftp that the cache is really cleared out.  But mostly I suspect there’s just a lag.

That help some?

Robin, thank you very much.  That helped clear up things greatly.  I did check clearing all cache, and it did work properly.  Your suggestion of a lag makes sense. 

I’m going to also increase the 1000 limit, since our servers have never had any disk i/o problems, but do have db connection problems, so hopefully that will balance things a bit better.

I do think there are still templating improvements I could make.  I am a a bit lost how to use the template debugging info that paul suggested:

you are talking about some tags taking over a second to process.  No matter the amount of traffic you are getting that is simply unacceptable for an EE tag to take that long.  Narrow down what tags or steps in those tags are taking the longest.  Create separate templates for those tags and try to solve each problem individually.

For example:

(0.004355) - Beginning Tag Processing -
(0.004373) Parsing Tags in Template
(0.004403) Tag: {exp:weblog:entries site="tmo|ipo" status="open|Featured|TMO|TMO Featured|iPO|iPO Featured" display_by="day" limit="6" disable="category_fields|member_data|pagination|trackbacks" dynamic="off" sticky="off" cache="yes" refresh="10"}
(0.004607) Closing Tag Found
(0.004711) Processing Tags
(0.004737) Module Tag: Weblog/entries
(0.004751) Including Files for Tag and Modules
(0.006076) Beginning Final Tag Data Processing
(0.006094) Calling Class/Method: Weblog/entries
(0.007665) -> Class Called: Weblog
(0.007735) -> Method Called: entries
(3.370225) Calling Extension Class/Method: Fieldframe/weblog_entries_tagdata
(3.370479) Calling Extension Class/Method: Tmo_forcetime/weblog_entries_tagdata

This is our homepage.  If I’m reading this debug information correctly, it takes 3 seconds to process the entries tag.  I’ve disabled as much as I can, and while our entries tag is complex, everything is crucial and necessary to create our homepage.  Paul’s advice is to figure out which tag is taking the longest, separate it out, and solve the problem individually, but I’m at a loss.  The tag is the entries tag… it just is telling me that the tag took 3 seconds.  How do I figure out which part of the tag is the worst offender, so I can start brainstorming alternative templating methods?

This is why we were looking for a caching solution… it make a night and day difference on many of our templates.  Super Search doesn’t help us because we are using FieldFrame and Primary Category.  We still have some dynamic section on our home-page, so Static Page Caching doesn’t work for us.  I’m hoping that Template Morsels helps us, but it seems odd to have to put the majority of our templates into morsels.  Is it worth asking for a feature that defines an absolute page/tag cache time, that overrides all other caching routines and flushing?

Hrm- looks like the extension is slow.  What I’d do first?  Put JUST the weblog tag on a blank page- no caching.  Then take a look at template debug again.  It will likely be slow.  But might as well double check.  If it is still slow- triple check you’re running the latest version of the extensions.  And… well, I’d disable all extensions and see if it’s STILL slow.  If it’s fast w/extensions disabled, slow with them enabled, you’ve at least narrowed in on the slow down.  And if it’s slow both ways- time to see why the weblog tag is taking so long.

Hrm- looks like the extension is slow.

Not necessarily.  The “Calling Extension Class/Method” log item is actually recorded *before* the calling of the extension’s method for that hook.  The only thing this tells us is that between the calling of the entries() method by the Template class and the calling of the extension, there is a time difference of ~3.4 seconds.  Anything between these two points could be causing the slow down.  What this does is narrow down where the problem may be.  The best way to debug this is to add additional $TMPL->log_item() calls inside the entries() method to see where the time is being taken up.  Exactly the sort of thing I would do during a Performance Evaluation.

Caching is a solution for the general load problem, yes, but it does not solve the problem of why processing of that tag is taking that abysmally long.  Knowing this could potentially allow more elegant solutions than trying to cache everything in Template Morsels.