Hi, after searching around the forums and web for some time, I still cannot find an answer to this question:
How do you set up your robots.txt file when you are running Multi Site Manager with your site on an addon domain, and are there directories people generally block? (im looking for an example). My system folder is above root level and the site i am looking to create robots for is not the MSM main site. There is not much in the folder, just:
- images
- templates (where I store my templates as files)
- themes (css, javascripts etc)
- user images
admin.php
index.php
htaccess
error_log
ftp_quota
Which files should I block?
I am wondering if i need to block things like member data - or does this no longer apply because my site is above root?
Any help on this would be greatly appreciated. I know what robots are so please no links to generic “abouts robots” pages please! 😊 Im sorry if this seems obvious, I just need some reassurance that I am doing it right.
This is what I have:
User-agent: *
Disallow: /templates
Disallow: /images
Disallow: /error-log
Disallow: /admin.php
Disallow: /themes
Disallow: /user-images
User-agent: Googlebot-Image
Disallow: /.
Disallow: /templates
Disallow: /images
Disallow: /error-log
Disallow: /admin.php
Disallow: /themes
Disallow: /user-images