Thread

I can’t turn the live site off.

Well, we have a bit of a problem here. We need a situation whereby we can reduce and isolate the problem.  This problem could stem from a ton of possibilities, as listed above.  You’re running a ton of extensions, including ones with a known historical problem such as the File extension, you may have template optimization issues (we haven’t even gone there yet), and you’re running a non-typical web server configuration using PHP as CGI.  Your environment may not be perfectly tuned for a dynamic database-driven system as well.

What we can do right now without touching your site is look at your templates (You can post one, such as your index, and we can have a quick look), ask your server administrator more about those errors in the server error log and why they’re occurring, and see where we get from there.

You can also make a complete backup of your site and put it on a development environment and see if you have speed problems, and do some more intensive testing there since it won’t affect the live site.

An option, of course, is to move to a known, excellent, hosting environment with administrators that are skilled and knowledgeable in tuning servers for high traffic, dynamic sites.  That would take out a major, possible contributing factor.

We have the two old VPS servers that were hosting the server before I moved it to the dedicated server yesterday. One is running PHP and Apache and the other is running MySQL. We can test with that.

The errors are being looked at too by the host.

Great, let us know what your test comes up with on those VPS servers, please!

I’m attaching our header, footer and home page index templates. There is a lot going on in the home page templates.

Right now with a slower load on the server, the home page is loading in under 10 seconds… this morning it was loading in 60+ seconds.

Anna

EDIT: template attachment removed

Alright; I think that the templates are going to end up being a huge part of your problem.  On your main index I counted, I believe, 19 weblog entries tags and that doesn’t include whatever you have in all of those embeds.  Furthermore, you have embeds within your weblog entries tags, so say you are showing 10 entries, that embed will occur 10 times.  Depending what’s in the embed, you could be really weighing the site down.

More concerning to me is that you have opened up some potential security issues in your templates.  You have code like like this:

{if "{segment_1}" == "seccion"}

See how your segment_3 variable is quoted and braced?  That is unnecessary and, since it can take user input, it is a security issue.  Please read Troubleshooting Conditionals to understand more about this.

At this point, I would recommend a complete template audit to optimize and figure out better ways of approaching the display of your data in a much less intensive way.

Anna,

I didn’t see any tag caching in the templates (but lots of disabling unneeded data, which is perfect) and not any mention of it in this thread, but I assume you’re caching the heck out of your templates?

And, yeah, there’s a ton going on with these templates. I’d concur with Lisa that they need to be re-evaluated and optimized. This may be a case where you don’t cache the entire template, but cache at the tag level for certain items that rarely change over the course of a day.

Right now with a slower load on the server, the home page is loading in under 10 seconds… this morning it was loading in 60+ seconds.

Yeah, with little load, you should be much, much faster than 10 seconds. Just so you know, MySQL will be your first bottleneck and CPU hog under heavy load. That’s why caching is so important.

Ryan has a point, most of our users come complain if page loads are over 1 second.  1.5 throws many of our forum users into panic.  10 seconds is… really indicative of a major underlying problem.  60 seconds is obvious, but 10 seconds is really not acceptable.

I agree we are having to use a lot of weblog tags to accomplish what this homepage needs to accomplish. But we are on a powerful dedicated server. It may not be tuned 100% but is sure seems like the performance should be better then what we are seeing right now. We are not having any memory issues at all, just CPU issues and MySQL is the largest consumer.

Yes, we are caching templates (homepage, header, styles, javascript) and have our MySQL configs tuned well.

This morning I went through the home page template and directly added any content that didn’t need to be in an embed and also added an {if logged_in} clause around our inline edit button embeds (this accounts for most of the embed tags). These changes do not appear to have made any difference.

One problem that has been identified is a lot of MySQL temp tables are being created on disk. The host let us know last night that BLOB and TEXT columns are not allow in memory tables and so are written to disk. We have 180 fields set up in this install and 90 of those are TEXT type columns since that is the EE default. Last night I change most of the 90 columns to varchar with a set character limit specific to the content. That apparently should have dramatically cut down on the % of temp tables being created on disk but I’m not seeing that this morning.

Also, yesterday Mitchell from Solspace went through the home page template with me with template debugging turned on and did not see anything that dramatically concerned him. We did upload a revised File extension that makes less calls to the database.

All told, we made a ton of changes yesterday and this morning that should have made a noticeably different but I’m not seeing any:(

Well, what I would do now, if you were game, would be to create a template with just Hello World in it and see if that’s slow.  Other processes on the server may slow this down, but you could certainly try it.

I would be that loads instantly.  So then would try a very simple weblog entries tag, and slowly add to it until you find the block.  Of course, if a plain template with some text in it takes 10 seconds to load (where other server processes aren’t affecting that) then we know its not your templates. =)

Here is the template
**This took 55 seconds for me to load
EDIT: Link to template removed

Also, here is a non ee php file
**This took 32 second to load for me
EDIT: Link to template removed

——————

HERE ARE MY TOP DETAILS

top - 10:51:07 up 1 day,  2:42,  1 user,  load average: 5.66, 5.34, 5.23
Tasks:  25 total,  1 running,  24 sleeping,  0 stopped,  0 zombie
Cpu0 : 63.1% us, 32.6% sy,  0.0% ni,  4.4% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu1 : 60.5% us, 36.4% sy,  0.0% ni,  3.1% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu2 : 65.8% us, 29.2% sy,  0.0% ni,  5.1% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu3 : 79.5% us, 18.9% sy,  0.0% ni,  1.3% id,  0.3% wa,  0.0% hi,  0.0% si
Mem:  9338156k total,  701288k used,  8636868k free,      0k buffers
Swap:      0k total,      0k used,      0k free,      0k cached

  PID USER     PR NI VIRT RES SHR S %CPU %MEM   TIME+  COMMAND
3320 mysql   15   0 621m 99m 4936 S 205 1.1 755:15.13 mysqld
27738 nobody   16   0 31152 12m 3400 S   53 0.1   1:37.62 php-cgi
26609 nobody   16   0 31756 12m 3396 S   40 0.1   2:39.84 php-cgi
27669 nobody   15   0 30468 11m 3400 S   28 0.1   2:06.17 php-cgi
27739 nobody   16   0 31000 12m 3376 S   28 0.1   1:18.68 php-cgi
27720 nobody   16   0 31784 12m 3400 S   27 0.1   1:33.55 php-cgi
1936 nobody   15   0 4208 2744 680 S   1 0.0   1:37.60 nginx
  1 root     16   0 1980 644 556 S   0 0.0   0:00.52 init
19802 root     16   0 6908 1048 672 S   0 0.0   0:01.73 sshd
19813 root     19   0 2636 856 692 S   0 0.0   0:00.00 xinetd
19825 root     16   0 4996 508 248 S   0 0.0   0:00.00 vsftpd
19948 root     16   0 8948 1836 772 S   0 0.0   0:00.66 sendmail
19956 smmsp   16   0 8048 1472 620 S   0 0.0   0:00.00 sendmail
19967 root     16   0 3204 1104 572 S   0 0.0   0:00.05 crond
19977 root     18   0 5396 688 424 S   0 0.0   0:00.00 saslauthd
19978 root     18   0 5396 432 168 S   0 0.0   0:00.00 saslauthd
27668 nobody   16   0 24488 8116 5536 S   0 0.1   0:00.15 php-cgi
1935 root     19   0 2276 408 220 S   0 0.0   0:00.00 nginx
1937 nobody   15   0 4028 2656 680 S   0 0.0   1:28.00 nginx
3260 root     17   0 2324 1068 912 S   0 0.0   0:00.00 mysqld_safe
5364 root     16   0 9944 2800 2264 S   0 0.0   0:01.36 sshd
5387 root     15   0 2380 1360 1100 S   0 0.0   0:00.01 bash
5512 root     15   0 9940 2860 2248 S   0 0.0   0:00.24 sshd
5514 root     15   0 6632 1724 1156 S   0 0.0   0:00.04 sftp-server
24304 root     16   0 2116 1020 820 R   0 0.0   0:02.34 top

Loading the pages now just a few minutes later is taking about 7 second on my end.

Also, here is a non ee php file
**This took 32 second to load for me
http://www.zocalo.com.mx/anna.php

This is strongly indicative of something server side, perhaps between MySQL and PHP.  You’re getting these results outside of ExpressionEngine, so I would strongly recommend taking that information back to your host.

Please do keep us updated.

What were your load times on those pages Lisa?

The first time very slow - so slow I kind of closed the tab.  The second and third time 3-5 seconds.

Your PHP example is between 1 and 5 seconds depending on the refresh.

Your main EE template took about 5 seconds to start loading, and another 5-7 seconds to render all the graphics, scripts, and images (that is after EE delivers the data, so that maybe could use a look at too?)

So this seems like a fastish time of day for your site.