How much important is to have robots.txt file?

jiten702's Avatar
Go4Expert Member
Hi all......


As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?
shabbir's Avatar, Join Date: Jul 2004
Go4Expert Founder
No it does not help in crawling but it helps in blocking Googlebot
nabeelarkisdata's Avatar, Join Date: Jun 2010
Go4Expert Member
Quote:
Originally Posted by shabbir View Post
No it does not help in crawling but it helps in blocking Googlebot
And what does that mean please? if googlebot are blocked, what happens?
shabbir's Avatar, Join Date: Jul 2004
Go4Expert Founder
Quote:
Originally Posted by nabeelarkisdata View Post
And what does that mean please? if googlebot are blocked, what happens?
The pages won't get indexed.
rogerf265's Avatar, Join Date: Aug 2010
Newbie Member
Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.

Last edited by shabbir; 2Aug2010 at 12:47.. Reason: Confine Links to signature only
shabbir's Avatar, Join Date: Jul 2004
Go4Expert Founder
It is robots.txt and not robot.txt
365outsource's Avatar, Join Date: Sep 2010
Newbie Member
When a crawler comes to your site, it will look for a special file on your site, called robots.text tells the bot, which Web pages of your site should be indexed and which Web pages should be ignored.
infoway USA's Avatar, Join Date: Jun 2010
Go4Expert Member
It has nothing to do with your SEO plan and strategy. Since it can;t increase ranking, visitors or whatever. but wold you want your personal data becomes public or any valuable document present on the web? It helps you keep crawler keep at bay from those pages you don't want to be crawled.
4deewar's Avatar, Join Date: Sep 2010
Go4Expert Member
robots.txt is use for tell the google that which page is index or which is not.
Click SSL's Avatar, Join Date: Mar 2010
Skilled contributor
Quote:
Originally Posted by shabbir View Post
No it does not help in crawling but it helps in blocking Googlebot
Still I am not getting perfectly how it helps in blocking?