Go4Expert (http://www.go4expert.com/)
-   Internet Marketing (http://www.go4expert.com/articles/internet-marketing/)
-   -   Robots.txt Files (http://www.go4expert.com/articles/robotstxt-files-t16034/)

PradeepKr 10Sep2010 15:34

Re: Robots.txt Files
Few cents from me,
You can add sitemap location also in the robots.txt file.

This would tell robots where exactly your sitemap is and which links to crawl.

parryrater 14Sep2010 16:44

Re: Robots.txt Files

really robot.txt file has more useful to make user friendly site..!!! meet again.

raantint 17Sep2010 12:24

Re: Robots.txt Files

Is there any robot.txt generator tool like as sitemap generator...??? meet again.

shabbir 17Sep2010 13:51

Re: Robots.txt Files
Visit Google Webmaster tool and they have option for generating one for your domain

parrytint 20Sep2010 12:32

Re: Robots.txt Files

What are the benefits by the robots.txt file generate..??? meet again.

raanzen 23Sep2010 11:34

Re: Robots.txt Files

Which URL's may not eligible to crawl by search engine in any any site...??? meet again.

vimlesh 24Feb2011 11:30

Re: Robots.txt Files
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.

delhifirm 4Mar2011 11:21

Re: Robots.txt Files
Thank for sharing.
It is a basic notes on Robots.txt Files that is good

denishverma 4Mar2011 15:49

Re: Robots.txt Files
Used to give authentication to Google bot or search engine bot for website pages and other folders.
Robots.txt file is a simple text file where we gives authentication for whole website inner folders.
Robots.txt allow to bot of every search engine.
User agent of Robots.txt is used to do this.

stacey 1Jun2011 16:56

Re: Robots.txt Files
Yep the robots.txt file useful for the website to protect the personal information from the public.

All times are GMT +5.5. The time now is 00:21.