Few cents from me, You can add sitemap location also in the robots.txt file. This would tell robots where exactly your sitemap is and which links to crawl.
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.
Robots.txt Used to give authentication to Google bot or search engine bot for website pages and other folders. Robots.txt file is a simple text file where we gives authentication for whole website inner folders. Robots.txt allow to bot of every search engine. User agent of Robots.txt is used to do this.
Thanks for the detailed information and moreover Robots.txt will follow both nofollow and dofollow links We cant give command to the Google robots to follow the needed links because basically robot will know all the linking process for the website promotion.
I have seen that Google tends to ignore robots.txt file sometimes. Pages which you have specified in robots as "Disallow" are sometimes seen crawled by Google.
Robots.txt is an authentication file which used to allow/ disallow folders of website. If disallow then anyone can not access the folder. If allows then can access folders, inner folders, sub folders or cgi_bin etc. thanks
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. User-agent: * Disallow: / The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
