Excluding IPs from Throttling?

Posted: 23 January 2006 05:30 PM

[ Ignore ]

Lab Assistant

Total Posts: 201

Joined 09-30-2004

I recently got a message from Google telling me that their spiders were having trouble on my site because they were being shut down and were getting a mesage that “You have exceeded the allowed page load frequency” They blamed my server, but I just found that it’s actually the throttling on EE.

(This was not some spammer’s trick—Google’s tech told me this as a side issue to an adsense question I had)

Is there a way to permit a range of ip addresses to go through? They gave me a range of ip addresses to allow.

Profile

Lisa Wess

Posted: 23 January 2006 05:41 PM

[ Ignore ] [ # 1 ]

Moderator

Total Posts: 32921

Joined 05-14-2004

I actually asked this during beta (I kept getting throttled ) and I believe the answer was done that the throttling is done first, before EE even parses the page, so that IP exclusion can’t be done. But I’ll double check and see if my memory is faulty.

The throttling was actually initially put in, I believe, because of overzealous bots, at that….

Signature

Profile

MSG

Paul Burdick

Posted: 23 January 2006 06:12 PM

[ Ignore ] [ # 2 ]

Research Scientist

Total Posts: 7534

Joined 08-05-2002

The IP of the the request is instantly available through server variables and the throttle check is actually done after we sanitize variables. However, there is not a built in way to give certain IPs a pass. If you want to hack the files, here it is:

Open up core.system.php and find this:

if ($PREFS->ini('enable_throttling') == 'y' AND REQ == 'PAGE')
     {
          require PATH_CORE.'core.throttling'.EXT;    
          
          $THR = new Throttling();
          $THR->throttle_ip_check();
          $THR->throttle_check();
          $THR->throttle_update();
     } 

What we are going to do is create an array of IP addresses and if an IP is not found in that array, then it does the throttle check:

if ($PREFS->ini('enable_throttling') == 'y' AND REQ == 'PAGE')
     {
          // GoogleBot (range), MSNBot (full IP)
          $pass = array('66.249.71.', '65.54.188.83');
          $skip = 'n';
          
          foreach($pass as $value)
          {
               if (strstr($IN->IP, $value))
               {
                    $skip = 'y';
               }
          }
     
          if ($skip == 'n')
          {
               require PATH_CORE.'core.throttling'.EXT;    
          
               $THR = new Throttling();
               $THR->throttle_ip_check();
               $THR->throttle_check();
               $THR->throttle_update();
          }
     } 

Signature

Profile

Lisa Wess

Posted: 23 January 2006 06:22 PM

[ Ignore ] [ # 3 ]

Moderator

Total Posts: 32921

Joined 05-14-2004

wiki’d. Thank you, Paul.

Signature

Profile

MSG

jretzer

Posted: 23 January 2006 06:36 PM

[ Ignore ] [ # 4 ]

Lab Assistant

Total Posts: 201

Joined 09-30-2004

It’s an interesting conundrum. On the one hand, you don’t want your system overwhelmed by requests. On the other, you DO want the search bots to fully index you.

I think I’m going to opt for full indexing.

Profile

‹‹ Typographic characters not being encoded in RSS feeds Global Variables not showing in Comments ››

Username		Remember Me?
Password		forgot password?

Post Marker Legend
	New posts	Hot Topic with new posts	New Poll	Moved Topic	Sticky topic
	No new posts	Hot Topic with no new posts	Old Poll	Closed Topic	Announcements

Visitor Statistics
The most visitors ever was 1149, on July 16, 2007 09:33 AM
Total Registered Members:	65074	Total Logged-in Users:	24
Total Topics:	82207	Total Anonymous Users:	13
Total Replies:	441811	Total Guests:	186
Total Posts:	524018