Hello, I think I have messed up big time all my websites. 6 months ago I decided to change the robots.txt files in 9 of my websites, a few weeks later I noticed a big drop in traffic(I thought it was the google updates), now 6 months later, none of them are indexed in Google, only bing. I am now suspecting that my robots.txt files were wrong. Could you please tell me is this code rejects Google bots please? User-agent: * Disallow: I have now replaced everything with: User-agent: * Disallow: /cgi-bin/ Thanks for your help everyone,
Go to Google Webmaster Tools, click on Health from the menu on the left and then click on Blocked URL's from the dropdown menu. This will show you how your robots.txt file is currently affecting your site, and you can test different variations of it to see what they do.
