Go4Expert (http://www.go4expert.com/)
-   Search Engine Optimization (SEO) (http://www.go4expert.com/forums/seo-forum/)
-   -   Robots.txt Suggestion (http://www.go4expert.com/forums/robotstxt-suggestion-t17340/)

jadsn 5May2009 09:46

Robots.txt Suggestion
Hello All,

I have my Robots.txt as


User-agent: *
Allow: /

But when looking at go4expert.com's robots.txt I see it has blocked many valid urls and so can anyone suggest for what and why

shabbir 5May2009 10:11

Re: Robots.txt Suggestion
robots.txt is not designed to tell what CAN be viewed, their point is to tell robots what they are NOT allowed to index. Search engines browse all pages by default so remove the "Allow: /" as it is invalid.

Also Go4Expert.com disallows pages where we can assume some spamming is done like member profile page

stephen186 5May2009 19:15

Re: Robots.txt Suggestion
I think rather than using "allow"...you should disallow as that is what you want to do if you want to hide some pages from crawlers.

All times are GMT +5.5. The time now is 19:02.