Something was hammering my server really bad and it seems that this part in my htaccess

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php/$1 [L]

is the reason, because when i take it out, server loads are normal…

Anything i can do about that? I would like to keep SE friendly urls if possible

also,

since i had to remove this code form my htaccess because my server load was to high all my links to category pages etc don’t work anymore

{category_title}

how do i have to change this now?

I haven’t seen other reports of that causing high loads, but if it’s causing a problem and you still want index.php removed, I’d try either the ‘Include’ or ‘Exclude List’ method laid out in the EE Wiki.  Switching over should enable you to clear up any issues with category links as well.

the other 2 methods didn’t work for some reason. It’s not that dramatic, I’ll just live with the index.php in the url and hope google doesn’t mind in future… But in my opinion EE shouldn’t keep the index.php in the url by default, I don’t understand why such a good script (best of it’s kind!!) has such a leak..

What you are using IS making a loop. that is why you are experiencing load problems. I mentioned before how to avoid such a loop in a thread, just remove slash from here:

/index.php/

Thanks EE Lover, just to clear this up,

THIS is the correct version, right?

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php/$1 [L]

what about

RewriteBase /

no need for this?

This is my completey htaccess, does anybody see another mistake or so?

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php/$1 [L]

RewriteEngine On
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?mydomain\.com/ [NC]
RewriteCond %{HTTP_REFERER} !^.*msn\. [NC]
RewriteCond %{HTTP_REFERER} !^.*yahoo\. [NC]
RewriteCond %{HTTP_REFERER} !^.*altavista\. [NC]
RewriteCond %{HTTP_REFERER} !^.*google\. [NC]
RewriteCond %{HTTP_REFERER} !search\?q=cache [NC]
RewriteRule \.(jpg|bmp|png)$ pictures/123.gif [L]

I don’t know much about rewrite rules, but here are some tips:

1.  generally I recommend removing your hotlink protection:

The problem is this anti-leech method relies on the http-referer code.  The referrer is sent by the client (browser).  That is the problem. Referrer is blocked by many firewalls and is not sent by many configurations.  So you may think you have stopped leeching problems, when what you have really done is block many people from seeing your website.

2.  check your error logs there should be such a report there:

mod_rewrite: maximum number of internal redirects reached. Assuming configuration error. Use ‘RewriteOptions MaxRedirects’ to increase the limit

Probably that is why you were ffacing load problems.

3.  this is my htaccess file:

This is what it dose:

1. Removes index.php from your URLs.
2. Adds a trailing slash to all your URLs if there is not one already.
3. Forces all requested none WWW URLs to WWW ones.

Hope that helps =)

Ummm yeah, I would not stop using hotlink protection.  a) His hotlink protection seems to still allow empty referrers to see the images, not blocking anyone with a firewall that kills referrer information, and b) just ask a few of our clients that after being hammered by heavy hotlinking from Myspace and other locations, where a single of these clients that normally does about 2 gigs a month in site traffic had an extra 30 gigs added in about 7 days from Myspace hotlinking alone, much less a few extra gigs from a free blogging tool that does not allow file uploads… this was for *1* picture on this clients site!

Hotlink protection is still a very good thing.  Allowing blank referrers to see the images fixes the issue of people not being able to see your images if they use firewalls that block referrer information being shared.  Of course I still think the personal firewalls that remove referrer information do more damage than good in the long run, but thats just my opinion…

Nevin, if you let visitors with no referrer to steal your band width then what is hot linking for? If you are targeted to an attack they can simply hide their referrer and continue sucking the BW.

just ask a few of our clients that after being hammered by heavy hotlinking from Myspace and other locations, where a single of these clients that normally does about 2 gigs a month in site traffic had an extra 30 gigs added in about 7 days from Myspace hotlinking alone, much less a few extra gigs from a free blogging tool that does not allow file uploads… this was for *1* picture on this clients site!

Wow !!

Because hotlinking is something a web site does, not the web browser, so unless every visitor to that site is hiding their referrer information via a firewall then the hotlink protection works.  There really is little or nothing a person using someplace like Myspace can do to hide the referrer information and to them it would be more work than its worth.  They are going to go find someone else to hotlink to if something requires work, likely the reason they hotlinked in the first place. 

Hotlinking generally is not an attack, its usually people that want to display an image on a site they can’t upload to, or they don’t want to pay to house an image they linked to their profiles because they found it on cool image on a site somewhere else.  Out of all of the image hijacking I have seen since early 1994 when worked with a group setting up an early web server, 99.95% of them were hotlinked by lazy or cheap people, not people wanting to do direct damage to someone.

That one image was 100k, it was referenced on about 175 different Myspace web profiles as the person was using it in a profile section that shows up on all “friends” pages that link to your profile on Myspace.  The result? An average of 2,000 requests per hour for the image x 100k x 24 hours x 7 days was about 30+ gigs of traffic… just from a Myspace.com hotlink.  After adding hotlink protection while the requests were still there for about 2 weeks, there was almost no transmittal used for them so users traffic levels returned to normal levels instantly.  No reports of issues of visitors not seeing the clients images either.  The server overhead for a .htaccess image hotlink protection code can be much less than the overhead caused by being hotlinked

Not saying it works in all cases, but its better than nothing, thats for sure!

I need this hotlink protection. If I wouldn’T have one, I’d pay MUCH more on bandwith for sure, my site got fuskered very often already…

About this RewriteBase /

Right now my site works without this, would there still be a good reason to add this?

Ok, since i changed the three lines of code as told without the ‘/’ before the index my server loads are MUCH better.

But still high…

I have about 25-30k traffic on my site.  If i leave it like this and my site grows I could get problems again in future and then I would have to remove this code. This would mean I’d loose a lot of SE traffic. So, it would be cool to have a solution…

Does anybody here have this amount of traffic or more and no problems with this htaccess code?

The strange thing is the server loads are high only from time to time.

so fact is:

- server loads are high ONLY with this code in the htaccess BUT only from time to time.
- about 2-3 times a day loads go over 10, where 0-2 is nice.
- removing this code drops server loads immidiatly.

I wished somebody could help me…

EDIT: Doesn’t pMachine use this code for the entire site?

I’m happy that removing that slash helped you, but still I think you should check your error logs. Logs may give you a clue.

which error logs?

Your server’s error logs. Even on shared hosting plans they give you your own error logs.

hello,

my hosting partner watches my server very carfully, I may correct myself later but I’m sure there are no errors, he would have told me. It’s just that the server load is high from time to time and removing the code in the htaccess causes immidiate drop down of server loads…

But I had another Idea… how much recources does the search module eat? This is the only thing used by users ‘from time to time’ during the day and when the code is removed in the htaccess - it doesn’t work, this would explain that the server load is normal again then…

my hosting partner watches my server very carefully,

You had that slash in your htaccess file, so at least we now that there had to be some errors logged for sure. [ my 2 cents]

Ok, so i looked at my error.log file and saw some errors which didn’t have anything to do with EE. I still fixed them to be sure. It didn’t help.

I’m about to give up since the SE friendlier URL isn’t officially supported by EE anyway (although it seems that they use it themself?)

To demonstrate it deep to the code what’s going on, this is from SSH:

————————————
My server with the index.php and without the code in the htaccess:

lastst pid: 73554;  load averages:  0.60,  0.64,  2.76
133 processes: 1 running, 129 sleeping, 3 zombie
CPU states:  3.9% user,  0.0% nice,  6.6% system,  0.0% interrupt, 89.5% idle
Mem: 818M Active, 212M Inact, 200M Wired, 86M Cache, 112M Buf, 678M Free
Swap: 4000M Total, 24K Used, 4000M Free

————————————
My server without the index.php and with the code in the htaccess:

last pid: 74678;  load averages:  9.23, 11.23,  7.54    
192 processes: 3 running, 185 sleeping, 4 zombie
CPU states: 82.6% user,  0.0% nice, 15.5% system,  1.9% interrupt,  0.0% idle
Mem: 1116M Active, 222M Inact, 206M Wired, 86M Cache, 112M Buf, 365M Free
Swap: 4000M Total, 24K Used, 4000M Free
———————————-

I guess nobody can help me now anymore :-(

I guess nobody can help me now anymore :-(

Not me, but Nevin is an expert, maybe he will get involved if he has the free time.