rebeccaasmit's Avatar
Go4Expert Member
Thanks for posting this !! A well apt complete knowledge of robots.txt.....
benivolentsoft's Avatar, Join Date: Feb 2011
Go4Expert Member
Thanks for the detailed information and moreover Robots.txt will follow both nofollow and dofollow links

We cant give command to the Google robots to follow the needed links because basically robot will know all the linking process for the website promotion.
seo-marketing's Avatar, Join Date: Jun 2009
Light Poster
I have seen that Google tends to ignore robots.txt file sometimes. Pages which you have specified in robots as "Disallow" are sometimes seen crawled by Google.
tiwvinay's Avatar
thanks for given information.
seoforums85's Avatar
When you do not want to crawl the page then use robot.txt file
castorsandwheels's Avatar
Light Poster
Thanks for the information about Robot.txt.
Creativepromotion's Avatar, Join Date: Dec 2010
Light Poster
useful info for everyone
denishverma's Avatar, Join Date: Aug 2010
Denish Verma- SEO Expert
Robots.txt is an authentication file which used to allow/ disallow folders of website.
If disallow then anyone can not access the folder.
If allows then can access folders, inner folders, sub folders or cgi_bin etc.

jhon786's Avatar, Join Date: Oct 2011
Go4Expert Member
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
mukeshsoftona's Avatar
Disallow or allow you website content from Google.