1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

It will work for stopping google bots?

Discussion in 'Internet Marketing' started by davidgaffer, Dec 24, 2012.

  1. davidgaffer

    davidgaffer New Member

    Joined:
    Dec 21, 2012
    Messages:
    12
    Likes Received:
    1
    Trophy Points:
    0
    Home Page:
    Hello,

    I think I have messed up big time all my websites.
    6 months ago I decided to change the robots.txt files in 9 of my websites, a few weeks later I noticed a big drop in traffic(I thought it was the google updates), now 6 months later, none of them are indexed in Google, only bing.

    I am now suspecting that my robots.txt files were wrong.

    Could you please tell me is this code rejects Google bots please?

    User-agent: *
    Disallow:

    I have now replaced everything with:
    User-agent: *
    Disallow: /cgi-bin/


    Thanks for your help everyone,
     
  2. ozsubasi

    ozsubasi New Member

    Joined:
    Jan 26, 2012
    Messages:
    505
    Likes Received:
    29
    Trophy Points:
    0
    Occupation:
    Real Estate
    Location:
    Altinkum, Didim, Turkey
    Home Page:
    Go to Google Webmaster Tools, click on Health from the menu on the left and then click on Blocked URL's from the dropdown menu.
    This will show you how your robots.txt file is currently affecting your site, and you can test different variations of it to see what they do.
     

Share This Page