What Code stop SE crawl a site?
|
Go4Expert Member
|
|
| 20Jul2010,14:56 | #1 |
|
what is the code through which I can ask any search engine not to crawl my own site or part of the site?
|
|
Go4Expert Founder
|
![]() |
| 20Jul2010,15:16 | #2 |
|
Multiple ways.
1. In robots.txt add the following line User-agent: * Disallow: / 2. In Head Meta you can specify no crawl or index like <meta name="robots" content="noindex,nofollow" /> 3. Last but by no means least is make it a password protect directories using Apache |
|
Go4Expert Member
|
|
| 28Jul2010,14:34 | #3 |
|
Disallow: / this is the code. this prevents search engines from crawling your site
|
