Thread

I am hoping that somebody can help us out here. I’m working on a large EE2 project and we’ve encountered a very critical issue which has us stumped.

The server environment, which is managed internally, is this: Load Balancer | Firewall | -> Reverse Proxy Server -> EE Server -> MySQL Server (a diagram of the server environment is attached). What is happening is that when any request is sent to the reverse proxy we are intermittently / sporadicly / randomly experiencing server timeouts. After sifting through server logs, we’ve found numerous examples of the error - it’s returning an Internal Server error (500). Here is a snippet from the error log (with site specific info removed):

192.168.X.XX - - [02/Nov/2010:14:24:54 -0400] “POST /system/index.php?S=0&D=cp&C=content_publish&M=entry_form HTTP/1.1” 500 1295 “http://my-dev.xx-xx.com/system/index.php?S…; “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7”]

We know that it has *something* to do with the reverse proxy, as we’ve bypassed it (hitting the box directly) and have eliminated the problem. There’s something between the proxy and the EE box that is causing these problems.

Please note that this happens in the control panel when submitting content and/or content changes as well as on the front end when sending any POST data - search, user login, or any custom forms. There is absolutely no rhyme nor reason to when it happens or why - when experiencing the timeout, I can hit the submit/publish button several times and it will eventually submit correctly. It’s also browser/platform agnostic - PC, Mac, Chrome, FF, Safari, etc. It’s occurring in multiple locations as well.

Does anybody have any thoughts as to why this would be happening? Any idea, no matter how off the wall, would be appreciated as we’ve run out of them.

We’ve come across this post in the CodeIgniter forums: http://ellislab.com/forums/viewthread/150496/#735762 - we’ve got the server admins tasked to commenting any of those calls out right now, but we don’t know if that will do anything.

Thanks in advance for anybody’s brainstorm!

Sam

How are you resolving IP addresses behind the proxy? Does it send an HTTP_X_FORWARDED_FOR header?

Hi, thanks for the reply.

the server team is currently working on implementing that.

right now all traffic comes from the reverse proxy IP with no http-x-forwarded-for

Sam

Ok, when you do, remember to add a proxy ips config setting =) .

This is happening with all requests, correct? Not just POST’s? Does it happen to guests viewing the frontend?

And lastly, do you have throttling enabled?

The proxy_ips config line is set correctly.

This is only happening when POSTing to the database. It is happening on both the front end of the site (via custom forms) and when either publishing/editing content. Other database transactions seem fine.

Throttling is *not* enabled. However, and I’m not sure if it matters, but the “Deny Access if No IP Address is Present” setting is set to Yes.

The hardest thing about this is that it’s seemingly random.

Do you have secure forms enabled?

The proxy_ips config line is set correctly.

It won’t actually pick it up until the forwarded for header is sent, but perfect!

However, and I’m not sure if it matters, but the “Deny Access if No IP Address is Present” setting is set to Yes.

That should only fire if absolutely no ip address is sent. What is happening right now is that every request gets the ip address of one of the proxies. So if the load balancer assigns you a different proxy on your POST request, then EE freaks out because your ip address changed. That would also explain why it appears to be random.

What is a little strange about it is that these 500s coming from EE are sent with a message body. You’re saying it simply hangs. However, that might be because they are caught and logged by one on the servers on the way out - instead of showing for the user.

secure_forms is turned off.

We currently only have one of the reverse proxies running (they apparently shut the other down to troubleshoot and it did not solve anything). The proxy_ip is set to the sole IP of the active proxy.

The system will just hang - the activity wheel in Chrome, for instance, will just spin for a while and will eventually result in an Apache/Tomcat 404 error - though according to the server logs, it’s actually throwing a 500.

Let me elaborate on that. From a member of the dev team:

“the reverse proxy returns a 404 but thats only because Expressionengine doesn’t return the resource it requested, but instead returned a 500.”

Aye, but ExpressionEngine returns a 500 with an error message in the message body.

Seeing that error would help pinpoint where it’s flaking out. We can remove the header to troubleshoot. Open up system/expressionengine/core/EE_Exceptions.php, on line 40, we call set_status_header(500). If you remove / comment out that line it should fail with a 200 - and hopefully give you a more descriptive error.

The server admin is tied up for the remainder of the day but will be making the change first thing in the morning. I hope to have additional info then.

Thanks for you help thus far!

Sam

Let us know what you find out. We’ll be here.

So close, yet so far…

Commenting that line out did indeed return a 200, but unfortunately, the server is truncating the line. We’re currently only seeing this:

writev(11, [{"HTTP/1.1 200 OK\r\nDate: Tue, 09 N"..., 220}, {"<html>\n<head>\n<title>Error</titl"..., 1236}], 2) = 1456

in the stack trace. The server folk are looking into making the stack trace more verbose so I’ll supply updates as the come flying in.

Thanks
Sam

Thanks for the continued updates Sam. Pascal will be working today and will hopefully chime in here again. Thanks for your patience

[ ignore this ]

We’ve got the error.

Wed Nov 10 10:08:19 EST 2010
Heading : An Error Was Encountered
Message : You are not authorized to access this page
Status Code: 500

Wed Nov 10 10:10:06 EST 2010
Heading : An Error Was Encountered
Message : You are not authorized to access this page
Status Code: 500