PradeepKr's Avatar
Go4Expert Member
Few cents from me,
You can add sitemap location also in the robots.txt file.

This would tell robots where exactly your sitemap is and which links to crawl.
parryrater's Avatar, Join Date: Aug 2010
Light Poster

really robot.txt file has more useful to make user friendly site..!!! meet again.
raantint's Avatar
Go4Expert Member

Is there any robot.txt generator tool like as sitemap generator...??? meet again.
shabbir's Avatar, Join Date: Jul 2004
Go4Expert Founder
Visit Google Webmaster tool and they have option for generating one for your domain
parrytint's Avatar
Go4Expert Member

What are the benefits by the robots.txt file generate..??? meet again.
raanzen's Avatar
Light Poster

Which URL's may not eligible to crawl by search engine in any any site...??? meet again.
vimlesh's Avatar
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.
delhifirm's Avatar, Join Date: Mar 2011
Light Poster
Thank for sharing.
It is a basic notes on Robots.txt Files that is good
denishverma's Avatar, Join Date: Aug 2010
Denish Verma- SEO Expert
Used to give authentication to Google bot or search engine bot for website pages and other folders.
Robots.txt file is a simple text file where we gives authentication for whole website inner folders.
Robots.txt allow to bot of every search engine.
User agent of Robots.txt is used to do this.
stacey's Avatar, Join Date: Jan 2011
Yep the robots.txt file useful for the website to protect the personal information from the public.