Go4Expert

Go4Expert (http://www.go4expert.com/)
-   Search Engine Optimization (SEO) (http://www.go4expert.com/forums/seo-forum/)
-   -   How much important is to have robots.txt file? (http://www.go4expert.com/forums/robotstxt-file-t22555/)

jiten702 26Jun2010 12:07

How much important is to have robots.txt file?
 
Hi all......


As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?

shabbir 26Jun2010 16:40

Re: How much important is to have robots.txt file?
 
No it does not help in crawling but it helps in blocking Googlebot

nabeelarkisdata 28Jul2010 15:09

Re: How much important is to have robots.txt file?
 
Quote:

Originally Posted by shabbir (Post 69653)
No it does not help in crawling but it helps in blocking Googlebot

And what does that mean please? if googlebot are blocked, what happens?

shabbir 28Jul2010 18:03

Re: How much important is to have robots.txt file?
 
Quote:

Originally Posted by nabeelarkisdata (Post 70851)
And what does that mean please? if googlebot are blocked, what happens?

The pages won't get indexed.

rogerf265 2Aug2010 12:08

Re: How much important is to have robots.txt file?
 
Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.

shabbir 9Aug2010 19:43

Re: How much important is to have robots.txt file?
 
It is robots.txt and not robot.txt

365outsource 7Sep2010 12:49

Re: How much important is to have robots.txt file?
 
When a crawler comes to your site, it will look for a special file on your site, called robots.text tells the bot, which Web pages of your site should be indexed and which Web pages should be ignored.

infoway USA 22Sep2010 17:48

Re: How much important is to have robots.txt file?
 
It has nothing to do with your SEO plan and strategy. Since it can;t increase ranking, visitors or whatever. but wold you want your personal data becomes public or any valuable document present on the web? It helps you keep crawler keep at bay from those pages you don't want to be crawled.

4deewar 25Sep2010 17:29

Re: How much important is to have robots.txt file?
 
robots.txt is use for tell the google that which page is index or which is not.

Click SSL 6Jun2011 14:15

Re: How much important is to have robots.txt file?
 
Quote:

Originally Posted by shabbir (Post 69653)
No it does not help in crawling but it helps in blocking Googlebot

Still I am not getting perfectly how it helps in blocking? :confused:


All times are GMT +5.5. The time now is 04:42.