Robots.txt Suggestion

jadsn's Avatar
Light Poster
Hello All,

I have my Robots.txt as

User-agent: *
Allow: /
But when looking at's robots.txt I see it has blocked many valid urls and so can anyone suggest for what and why
shabbir's Avatar, Join Date: Jul 2004
Go4Expert Founder
robots.txt is not designed to tell what CAN be viewed, their point is to tell robots what they are NOT allowed to index. Search engines browse all pages by default so remove the "Allow: /" as it is invalid.

Also disallows pages where we can assume some spamming is done like member profile page
stephen186's Avatar, Join Date: Feb 2009
Go4Expert Member
I think rather than using "allow" should disallow as that is what you want to do if you want to hide some pages from crawlers.