Go4Expert (http://www.go4expert.com/)
-   Internet Marketing (http://www.go4expert.com/articles/internet-marketing/)
-   -   Robots.txt Files (http://www.go4expert.com/articles/robotstxt-files-t16034/)

rebeccaasmit 7Jun2011 13:08

Re: Robots.txt Files
Thanks for posting this !! A well apt complete knowledge of robots.txt.....

benivolentsoft 14Jun2011 16:38

Re: Robots.txt Files
Thanks for the detailed information and moreover Robots.txt will follow both nofollow and dofollow links

We cant give command to the Google robots to follow the needed links because basically robot will know all the linking process for the website promotion.

seo-marketing 17Jun2011 13:41

Re: Robots.txt Files
I have seen that Google tends to ignore robots.txt file sometimes. Pages which you have specified in robots as "Disallow" are sometimes seen crawled by Google.

tiwvinay 14Jul2011 17:11

Re: Robots.txt Files
thanks for given information.

seoforums85 29Jul2011 15:24

Re: Robots.txt Files
When you do not want to crawl the page then use robot.txt file

castorsandwheels 4Aug2011 17:11

Re: Robots.txt Files
Thanks for the information about Robot.txt.

Creativepromotion 12Aug2011 18:04

Re: Robots.txt Files
useful info for everyone

denishverma 15Aug2011 07:22

Re: Robots.txt Files
Robots.txt is an authentication file which used to allow/ disallow folders of website.
If disallow then anyone can not access the folder.
If allows then can access folders, inner folders, sub folders or cgi_bin etc.


jhon786 12Oct2011 11:21

Re: Robots.txt Files
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

mukeshsoftona 15Nov2011 15:38

Re: Robots.txt Files
Disallow or allow you website content from Google.

All times are GMT +5.5. The time now is 17:39.