It will work for stopping google bots?

davidgaffer's Avatar, Join Date: Dec 2012
Go4Expert Member

I think I have messed up big time all my websites.
6 months ago I decided to change the robots.txt files in 9 of my websites, a few weeks later I noticed a big drop in traffic(I thought it was the google updates), now 6 months later, none of them are indexed in Google, only bing.

I am now suspecting that my robots.txt files were wrong.

Could you please tell me is this code rejects Google bots please?

User-agent: *

I have now replaced everything with:
User-agent: *
Disallow: /cgi-bin/

Thanks for your help everyone,
ozsubasi's Avatar, Join Date: Jan 2012
Invasive contributor
Go to Google Webmaster Tools, click on Health from the menu on the left and then click on Blocked URL's from the dropdown menu.
This will show you how your robots.txt file is currently affecting your site, and you can test different variations of it to see what they do.