I need someone to give me a general run down on how to setup a robots.txt for my expression engine site. Can someone give me an example that I can use as a guide?
Also, I see in other threads that EE staff recommend telling the googlebots not to spider your system folder - but that doesn’t that require you naming your system folder? Isn’t that dangerous?
I am using MSM, and need to setup robots.txt for main site and additional site.
Thanks!