1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How much important is to have robots.txt file?

Discussion in 'Search Engine Optimization (SEO)' started by jiten702, Jun 26, 2010.

  1. jiten702

    jiten702 New Member

    Joined:
    Jun 24, 2010
    Messages:
    33
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    [FONT=Garamond, Garamond, serif]Hi all......[/FONT]


    [FONT=Garamond, Garamond, serif]As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?[/FONT]
     
  2. shabbir

    shabbir Administrator Staff Member

    Joined:
    Jul 12, 2004
    Messages:
    15,285
    Likes Received:
    364
    Trophy Points:
    83
    No it does not help in crawling but it helps in blocking Googlebot
     
  3. nabeelarkisdata

    nabeelarkisdata New Member

    Joined:
    Jun 19, 2010
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    0
    Occupation:
    Webmaster
    Location:
    Mauritius
    Home Page:
    And what does that mean please? if googlebot are blocked, what happens?
     
  4. shabbir

    shabbir Administrator Staff Member

    Joined:
    Jul 12, 2004
    Messages:
    15,285
    Likes Received:
    364
    Trophy Points:
    83
    The pages won't get indexed.
     
  5. rogerf265

    rogerf265 New Member

    Joined:
    Aug 2, 2010
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.
     
    Last edited by a moderator: Aug 2, 2010
  6. shabbir

    shabbir Administrator Staff Member

    Joined:
    Jul 12, 2004
    Messages:
    15,285
    Likes Received:
    364
    Trophy Points:
    83
    It is robots.txt and not robot.txt
     
  7. 365outsource

    365outsource New Member

    Joined:
    Sep 7, 2010
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    When a crawler comes to your site, it will look for a special file on your site, called robots.text tells the bot, which Web pages of your site should be indexed and which Web pages should be ignored.
     
  8. infoway USA

    infoway USA New Member

    Joined:
    Jun 14, 2010
    Messages:
    34
    Likes Received:
    0
    Trophy Points:
    0
    Occupation:
    Webmaster
    Location:
    West Hills, CA, USA
    Home Page:
    It has nothing to do with your SEO plan and strategy. Since it can;t increase ranking, visitors or whatever. but wold you want your personal data becomes public or any valuable document present on the web? It helps you keep crawler keep at bay from those pages you don't want to be crawled.
     
  9. 4deewar

    4deewar New Member

    Joined:
    Sep 25, 2010
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Jaipur
    Home Page:
    robots.txt is use for tell the google that which page is index or which is not.
     
  10. Click SSL

    Click SSL Member

    Joined:
    Mar 8, 2010
    Messages:
    215
    Likes Received:
    5
    Trophy Points:
    18
    Location:
    Newark, DE, USA
    Home Page:
    Still I am not getting perfectly how it helps in blocking? :confused:
     
  11. shabbir

    shabbir Administrator Staff Member

    Joined:
    Jul 12, 2004
    Messages:
    15,285
    Likes Received:
    364
    Trophy Points:
    83
    If you add things like
    Code:
    User-agent: *
    Disallow: /
    
     
  12. shiv2011

    shiv2011 New Member

    Joined:
    Jun 3, 2011
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    0
    its very important because we exclude comtent that we donot want to crawl.
     
  13. shiv2011

    shiv2011 New Member

    Joined:
    Jun 3, 2011
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    0
    it exclude contents that we donot want to crawl.
     
  14. linkbuilding5

    linkbuilding5 Banned

    Joined:
    May 26, 2011
    Messages:
    62
    Likes Received:
    0
    Trophy Points:
    0
    robot.txt file is the file where you can allow or disallow a webpage to crawl.
     

Share This Page