Great info on robots.txt file. It is definently a tool that a lot of people don't utilise. Most likely its because they are not aware of it. Great post!

It is easy for robots/spiders to crawl your site if you have this file..

Good. can you show how to write when we want to say nofollow url?

one should also include sitemap path in the robots.txt file for better results.

hello bro,
follow me,you need the letter "p".you will see me in hackthis.co.uk
try to solve it.

We Use Robots Tag on all pages that we want indexed for the website. This will instruct the robots to crawl the page of the website.
Robot tag is implemented in following way:
<meta name="robots" content="index" />

great post

i got a vast discription

Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.

good article