Thread

Moderator’s note: Split off from here.

Forgive me if I should be starting a new thread, but I’m actually experiencing an identical problem on mediatemple right now.  And I’m already paying extra for the MySQL container!

Oddly enough all of my WordPress sites continue functioning properly during the time I get this error on three different EE sites.

Mediatemple said they could find nothing wrong and that I should contact ExpressionEngine support.  I tend to agree that it isn’t an EE problem, but it is a bit odd that the WordPress sites keep running.

What version and build of EE are you on? What error message are you getting, exactly? Where and when?

Thanks for the reply, Ingmar.  I am just now noticing that you started a new topic for me.

I am running EE 1.6.8
Build:  20090723

Often the site just hangs, “waiting for…” in the status bar of the browser.  When it does resolve (after minutes) it will be a 500 error or MySQL Error 2013.  Here are a couple examples.

MySQL ERROR:

Error Number: 2013

Description: Lost connection to MySQL server during query

Query: SELECT t.template_name, tg.group_name, t.edit_date FROM exp_templates t, exp_template_groups tg WHERE t.group_id = tg.group_id AND t.template_type = 'css' AND t.site_id = '1' AND ((t.template_name = 'main-css' AND tg.group_name = 'stylesheets') OR (t.template_name = 'ie6-css' AND tg.group_name = 'stylesheets') OR (t.template_name = 'ie6-7-css' AND tg.group_name = 'stylesheets'))

MySQL ERROR:

Error Number: 2013

Description: Lost connection to MySQL server during query

Query: SELECT * FROM exp_member_groups WHERE site_id = '1' AND group_id = '3'

Oddly, while this is happening, the WordPress sites I have on the same server are running just fine.  Also odd is that one of my EE sites has been running fine for about 10 months on this server and is just now having this trouble (I did not make any changes to it recently).  That particular site is running EE 1.6.7, Build 20090320

Final odd note is that I do not get MySQL errors from the control panel.

I am on mediatemple gridserver with a mysql container.  They seem to think everything is working fine, so I am in a bit of a bind.

Alex,

Sadly there isn’t a magic bullet for this. Take a look at this from the mySQL site.

Do you or can your host check your mysql logs for any more details?

Thanks, John.  I understand.  And thanks for the suggestion.  The MySQL container has a “slow query report” which I will run again (last time it came back with no slow queries).

I turned sql query caching off in my EE settings (saw that suggested in other recent threads about similar 2013 mediatemple problem) and it may have helped.  I haven’t seen the problem for about 24 hours myself, though an application (simon) that I use to monitor the site is registering a timeout every now and again.

Mediatemple suggested the following which I plan on doing as soon as I can find the time, or the next time the problem gets more frequent.

The best way to combat this issue is to click around on your site and view the processes that are being generated for each item you are executing on the site. The 500 internal server errors usually means you are hitting the PHP process limit on the Grid-Service. For information on CGI and PHP resource limits on the (gs) Grid-Service see our article at http://kb.mediatemple.net/questions/1620/ . If you click a certain part of your site, it most likely starts a PHP process which gathers MySQL information. Once the MySQL information is gathered, the PHP process will end. A new PHP process can not start for the same information, until the first one completes. If you click one section of the site a couple of times in rapid succession, the PHP processes can stack which can lead to overall performance issues.

You will need to use SSH with the “time” command to get indications of how long PHP processes are taking to complete. Then, you need to look at MySQL as well to get indications of how long the MySQL queries are taking. These times will give you the overall average a process is taking on the Grid-Service to gather information and execute. Also the query log provides great information on the MySQL environment, just be sure to run it after about 24hrs of the container be up after a reboot.

I’m not sure when or if I will have anything more info so you need to close this thread, please do.  If I ever figure anything out I may just post it in the community forum.

Thanks, again.

We can keep this one open for a little longer, I think… let us know if you find out anything else.

I did get a resolution to this.  It was an issue at mediatemple, which makes sense given the error, but I still don’t have a clear answer on why it was only happening to my EE sites (not other database-driven sites), and both at the same time.

I monitored it for a while and it seemed a pretty odd error. So in case it helps here is what I finally said to MT that prompted them to look into it and fix it, as well as their response explaining what the problem was.

I want to update this ticket with some new information that I hope will suggest to you what the problem might be.

I have an application that alerts me when the site takes longer than 30 seconds to load.  When I receive the alert I immediately check the site and will almost always be able to request several pages that fail to load in less than 60 seconds.  I have not seen a 500 error or a MySQL error 2013, but the pages are taking a very long time to load.

* Here is the strange part *
At the same time as I am experiencing slow page load I access these sites through a proxy and they load repeatedly with _no problems_.  I have also had someone in another state check the site while I am experiencing problems and they experienced no latency.  This is not an issue of the pages being poorly optimized as they load just fine most of the time (and load fine all of the time through a proxy).  It is also not an issue of my internet connection becoming slow across the board.  All other sites, including other database-driven sites on the same (mt) server, continue to run at full speed.

Does this suggest anything to you?

[MT RESPONSE]

The way you describe this sounds like a textbook “node” problem. This is really the only thing I can think of that fits the bill. The (gs) Grid-Service is setup in such a way that requests for your website are routed, at random, through one of many “nodes”, which are basically racked machines running the webserver, and connecting to our storage segments to display your site. This helps load-balance the Cluster and keep things running smoothly.

Occasionally, a node will crash and our admins will be alerted, who will then bring it back online or hide it from the load balancer. However, in this case, it looks like one of the nodes was accepting connections normally, but timing out when connecting to the database. This is why we were not automatically alerted; The monitoring looked normal. It also explains why you might get a stale connection (you’re hitting the bad node), but other connections are coming through jsut fine.

Our system admins have corrected the problematic node and I can confirm at this time that your site can be loaded from all of them quickly. Please do alert us again if you have any other problems,

Thanks for the support as I worked through this one.

Alex, a very specific problem and a fix from MT. Hopefully that did the trick. Feel free to start a new thread if you have any more questions.