I have my Robots.txt as
Code:
User-agent: * Allow: /
![]() |
|
|
#1 |
|
Light Poster
Join Date: May 2009
Posts: 5
Thanks: 0
Thanked 0 Times in 0 Posts
Rep Power: 0 ![]() |
Robots.txt Suggestion
Hello All, I have my Robots.txt as Code:
User-agent: * Allow: / |
|
|
|
|
|
#2 |
|
Go4Expert Founder
![]() |
Re: Robots.txt Suggestion
robots.txt is not designed to tell what CAN be viewed, their point is to tell robots what they are NOT allowed to index. Search engines browse all pages by default so remove the "Allow: /" as it is invalid. Also Go4Expert.com disallows pages where we can assume some spamming is done like member profile page |
|
|
|
|
|
#3 |
|
Go4Expert Member
Join Date: Feb 2009
Posts: 44
Thanks: 0
Thanked 2 Times in 1 Post
Rep Power: 0 ![]() |
Re: Robots.txt Suggestion
I think rather than using "allow"...you should disallow as that is what you want to do if you want to hide some pages from crawlers.
|
|
|
|
![]() |
|
More
|
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | Search this Thread |
| Display Modes | |
| Bookmarks | |
|
|
|
|
||||||||||||||||||||||||||||||||||||