[shabbir]

Quote:

Originally Posted by Click SSL

Still I am not getting perfectly how it helps in blocking?

If you add things like

User-agent: *
Disallow: /

[shiv2011]

its very important because we exclude comtent that we donot want to crawl.

[shiv2011]

it exclude contents that we donot want to crawl.

[linkbuilding5]

robot.txt file is the file where you can allow or disallow a webpage to crawl.