Quote:
Originally Posted by sandrajolly View Post
Robots.txt file does not improve your search engine positioning.
It provides robots with information concerning which files you will not allow to be crawled and indexed in the search engines.
When the search engine robot crawls your site it looks for the robots.txt file.
If it doesn't find one it assumes automatically that it may crawl and index the entire site.

This allows all robots to crawl all files.
User-agent: *
Disallow:

This Disallows all robots to crawl a folder called /cmsbuffet/ .
User-agent: *
Disallow: /cmsbuffet/
The clue to where this is copied from is in the reference to "cmsbuffet"
It originally comes from:
http://www.cmsbuffet.com/robots-txt-check.php