1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt Suggestion

Discussion in 'Search Engine Optimization (SEO)' started by jadsn, May 5, 2009.

  1. jadsn

    jadsn New Member

    Joined:
    May 5, 2009
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    0
    Hello All,

    I have my Robots.txt as

    Code:
     
    User-agent: *
    Allow: /
    
    But when looking at go4expert.com's robots.txt I see it has blocked many valid urls and so can anyone suggest for what and why
     
  2. shabbir

    shabbir Administrator Staff Member

    Joined:
    Jul 12, 2004
    Messages:
    15,293
    Likes Received:
    365
    Trophy Points:
    83
    robots.txt is not designed to tell what CAN be viewed, their point is to tell robots what they are NOT allowed to index. Search engines browse all pages by default so remove the "Allow: /" as it is invalid.

    Also Go4Expert.com disallows pages where we can assume some spamming is done like member profile page
     
  3. stephen186

    stephen186 New Member

    Joined:
    Feb 17, 2009
    Messages:
    43
    Likes Received:
    0
    Trophy Points:
    0
    I think rather than using "allow"...you should disallow as that is what you want to do if you want to hide some pages from crawlers.
     

Share This Page