|
Go4Expert Member
|
|
| 7Jun2011,13:08 | #31 |
|
Thanks for posting this !! A well apt complete knowledge of robots.txt.....
|
|
Go4Expert Member
|
|
| 14Jun2011,16:38 | #32 |
|
|
Thanks for the detailed information and moreover Robots.txt will follow both nofollow and dofollow links
We cant give command to the Google robots to follow the needed links because basically robot will know all the linking process for the website promotion. |
|
Light Poster
|
|
| 17Jun2011,13:41 | #33 |
|
|
I have seen that Google tends to ignore robots.txt file sometimes. Pages which you have specified in robots as "Disallow" are sometimes seen crawled by Google.
|
|
Banned
|
|
| 14Jul2011,17:11 | #34 |
|
|
thanks for given information.
|
|
Banned
|
|
| 29Jul2011,15:24 | #35 |
|
|
When you do not want to crawl the page then use robot.txt file
|
|
Light Poster
|
|
| 4Aug2011,17:11 | #36 |
|
|
Thanks for the information about Robot.txt.
|
|
Light Poster
|
|
| 12Aug2011,18:04 | #37 |
|
|
useful info for everyone
|
|
Denish Verma- SEO Expert
|
|
| 15Aug2011,07:22 | #38 |
|
Robots.txt is an authentication file which used to allow/ disallow folders of website.
If disallow then anyone can not access the folder. If allows then can access folders, inner folders, sub folders or cgi_bin etc. thanks |
|
Go4Expert Member
|
|
| 12Oct2011,11:21 | #39 |
|
|
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
User-agent: * Disallow: / The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. |
|
Banned
|
|
| 15Nov2011,15:38 | #40 |
|
|
Disallow or allow you website content from Google.
|