Thread

Hi All,

My site is a real-estate listing site, so the client will be regularly adding and deleting various listings on the site. I’m using the Structure module and I have the real-estate listings set up as Structure listings.

My client’s SEO person has told me to write a robots.txt file that excludes all the actual listings from being crawled. I’m not sure how to do this since those pages are generated dynamically and change regularly. Can anyone give me some guidance? Thanks!

Basically you will need to create a text file called robots.txt
You have to put this in the root directory of the site.

In it you just list which robots to disallow access to (* means all robots) and which directories/files to block from them.

Example

User-agent: *
Disallow: /listings/

This would stop all robots accessing any file within the “listings” template group.

Wouldn’t that keep search engines from crawling the “listings” directory at the root of the site? Shouldn’t it be “/system/expressionengine/templates/listings/”?

OK lets look at this a little differently.
Which template-group(s) and template(s) need to be block from the search engines?

The example I gave would stop search engines accessing anything beyond http://www.examplesite.com/listings/

This presumably contains the templates files you would find in the /system/expressionengine/templates/listings/ directory on the server?

I understand the concept, but wouldn’t writing a robots.txt file that just said “Disallow: /listings/” tell search engines not to crawl the directory named “listings” at the root of my site?

What if I had two directories named “listings” in two different places on my site? How would the search engine know which one to crawl and which one to not crawl?

So shouldn’t the robots.txt file look like this?:

User-agent: *
Disallow: /system/expressionengine/templates/listings/

Yep that’s right search engines wouldn’t crawl http://www.examplesite.com/listings/

I’m not sure I understand why/how there are 2 directories in the same place with the same name on your site…?
Can you write which specific directories or individual files/templates need to be blocked and we can work from there.

They’re not in the same place.

./listings/

is different from

./system/expressionengine/templates/listings/

One is at the root of the site, the other is in the templates folder.

However all this talk about template groups is moot because I’m using the Structure module, which bypasses the standard template system.

Ah yea I forgot that you mentioned using Structure.
Looking at it from a browsers point of view then, which urls should the spiders not be indexing?

http://www.mysite.com/find_a_community/community_profiles/example_community_1

Any listing after “community_profiles/”

This should probably do it then

User-agent: *
Disallow: /find_a_community/community_profiles/

Correct me if I’m wrong, but wouldn’t that instruct search engines not to index files in a directory called “community_profiles”? Obviously there’s no directory like that on the site, since the pages are generated dynamically.

Is there a way to write a robots.txt file using EE tags?

Hmmm good question. I’m not sure so if anyone else wants to chime in and clarify.