Go4Expert

Go4Expert (http://www.go4expert.com/)
-   Search Engine Optimization (SEO) (http://www.go4expert.com/forums/seo-forum/)
-   -   What Code stop SE crawl a site? (http://www.go4expert.com/forums/code-stop-se-crawl-site-t22765/)

infoway USA 20Jul2010 14:56

What Code stop SE crawl a site?
 
what is the code through which I can ask any search engine not to crawl my own site or part of the site?

shabbir 20Jul2010 15:16

Re: What Code stop SE crawl a site?
 
Multiple ways.

1. In robots.txt add the following line
User-agent: *
Disallow: /

2. In Head Meta you can specify no crawl or index like

<meta name="robots" content="noindex,nofollow" />

3. Last but by no means least is make it a password protect directories using Apache

nabeelarkisdata 28Jul2010 14:34

Re: What Code stop SE crawl a site?
 
Disallow: / this is the code. this prevents search engines from crawling your site


All times are GMT +5.5. The time now is 00:09.