Thread

Is having a robots.txt required?

Will not having one prevent my site from being crawled by search engines?

If this file is required, how should its contents look like?

User-agent: *
Disallow:

Is enough?

Thanks Victor.

Just curious. What happens if one doesnt include the robots.txt file on their root folder?

It is not a requirement to have to be indexed by search engines. But it is a good practice because you can block specific areas from being indexed by search engines.

So if I dont have this file, I wont get indexed by search engines?

It is not required. If you don’t have it, it will still get indexed.

robots.txt is a method to exclude well-behaved robots from certain parts of your site. If you want them to crawl everything, there is nothing you need to do.

Thanks for clarifying that out.

By the way, Ingmar, do you think you can help me out with this: http://ellislab.com/forums/viewthread/95135/