Thread

I have several pages I don’t want robots to crawl—my standalone entry page, for example. Right now Google has it.  So I’m trying to write a robots.txt file that will work well with Expression Engine.

What I’m confused about is whether I need the /index.php/ part in there, and also how to add Expression Engine pages that don’t end in .html or anything else.

For example, my standalone entry page is http://www.example.com/index.php/weblog/article_standalone_requests

How would I write that for the robots.txt?

And where do I put the robots.txt file?

I thought I was doing it wrong because I was getting no response through Google Webmaster Tools.  Turns out I have some conflicts between http://www.example.com and just example.com.  To Google they’re (apparently) very different. 

I think I have it sorted out. I’m doing robots.txt in the traditional way, but am cleaning up the site so the domain is consistent site-wide.

Narthex,

Along with making sure your URL is consistent with either www or not you could also remove the index.php