Thread

Hi,

Fast crawler has made the GET requests 298478 times since 11/Sep/2008:18:24:19 -0700 and it is causing the significant load increase on the server.

The EE cpanel/blogs publishing and blogs.cisco.com site are on the same server. Since they are on the same server, this might be impacting the blog posts publishing process slower.

So can the server load increase impact the Cpanel response time during blog posts publishing time.

Thanks.

I’ve moved this down to technical support for you.  Is Fast Crawler something you are running on purpose?

The brief answer to your question is that any crawler that is constantly hitting your site can definitely have an impact on performance.  This is seen with spam attacks, and things like the Slashdot and Digg effect.

If these are on the same server, share the same database, etc, then yes they can impact performance.

Thanks for the quick confirmation,Lisa. Yes, All are on same server.

The FAST search crawler is the internal search used by Cisco.  I have contacted the cisco team to make sure the crawler does not make that many requests.

I have blocked crawler for some time to avoid the performance issue.

That is probably your best option, especially if it is being a bit overzealous.  Your server team, whomever manages the crawler, should be able to tweak it. I’m not sure why it would need to crawl quite so ... eagerly. =)

Is there anything else relating to this that you need assistance with?

Right now I cannot think off any other questions related to this.

Thanks for the suggestions.

Hi,

I used the Awstats tool and who:Robots/Spiders visitors shows many hits by unknow bots.

So Pls tell me if any hits by unknown bots mean the spam bot hits.

Thanks.

You’d be best to discuss statistics directly with your server admin or someone who has experience with that software.  That is really wholly separate from ExpressionEngine itself.

Hi,

Please tell me if I can keep some code to plug in the template to find out how much time is the blogs.cisco.com is really taking load completely.

one of our blogger mentioned that :
There are a couple of comments that I cannot approve because it keeps timing out.

Please let me know how to fix this issue.

Thanks

If your server is getting hammered by spiders and crawlers, external or internal, this will definitely have an impact on performance. As Lisa suggested, talking to your server admin and/or checking server logs might be a good idea. No, not all unknown bots are automatically nefarious spammers. Perhaps you might want to read up on the robots.txt protocol to restrict the flow somewhat.

Hi Ingmar,

We have blocked the internal crawler that is impacting server performance. But I like to know if there is any code as asked in my previous reply.

Thanks

It sounds like you might want to enable Display Template Debugging.