Few cents from me,
You can add sitemap location also in the robots.txt file.
This would tell robots where exactly your sitemap is and which links to crawl.
|
Go4Expert Member
|
|
| 10Sep2010,15:34 | #21 |
|
|
Light Poster
|
|
| 14Sep2010,16:44 | #22 |
|
|
hi...
really robot.txt file has more useful to make user friendly site..!!! meet again. |
|
Go4Expert Member
|
|
| 17Sep2010,12:24 | #23 |
|
|
hi...
Is there any robot.txt generator tool like as sitemap generator...??? meet again. |
|
Go4Expert Founder
|
 |
| 17Sep2010,13:51 | #24 |
|
Visit Google Webmaster tool and they have option for generating one for your domain
|
|
Go4Expert Member
|
|
| 20Sep2010,12:32 | #25 |
|
|
hi...
What are the benefits by the robots.txt file generate..??? meet again. |
|
Light Poster
|
|
| 23Sep2010,11:34 | #26 |
|
|
hi...
Which URL's may not eligible to crawl by search engine in any any site...??? meet again. |
|
Banned
|
|
| 24Feb2011,11:30 | #27 |
|
|
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.
|
|
Light Poster
|
|
| 4Mar2011,11:21 | #28 |
|
|
hello
Thank for sharing. It is a basic notes on Robots.txt Files that is good |
|
Denish Verma- SEO Expert
|
|
| 4Mar2011,15:49 | #29 |
|
Robots.txt
Used to give authentication to Google bot or search engine bot for website pages and other folders. Robots.txt file is a simple text file where we gives authentication for whole website inner folders. Robots.txt allow to bot of every search engine. User agent of Robots.txt is used to do this. |
|
Contributor
|
|
| 1Jun2011,16:56 | #30 |
|
|
Yep the robots.txt file useful for the website to protect the personal information from the public.
|