1. We have moved from vBulletin to XenForo and you are viewing the site in the middle of the move. Though the functional aspect of everything is working fine, we are still working on other changes including the new design on Xenforo.
    Dismiss Notice

Robots.txt Files

Discussion in 'Internet Marketing' started by CircuitX, Feb 2, 2009.

  1. PradeepKr

    PradeepKr New Member

    Few cents from me,
    You can add sitemap location also in the robots.txt file.

    This would tell robots where exactly your sitemap is and which links to crawl.
  2. parryrater

    parryrater New Member


    really robot.txt file has more useful to make user friendly site..!!! meet again.
  3. raantint

    raantint New Member


    Is there any robot.txt generator tool like as sitemap generator...??? meet again.
  4. shabbir

    shabbir Administrator Staff Member

    Visit Google Webmaster tool and they have option for generating one for your domain
  5. parrytint

    parrytint New Member


    What are the benefits by the robots.txt file generate..??? meet again.
  6. raanzen

    raanzen New Member


    Which URL's may not eligible to crawl by search engine in any any site...??? meet again.
  7. vimlesh

    vimlesh Banned

    The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.
  8. delhifirm

    delhifirm New Member

    Thank for sharing.
    It is a basic notes on Robots.txt Files that is good
  9. denishverma

    denishverma Denish Verma- SEO Expert

    Used to give authentication to Google bot or search engine bot for website pages and other folders.
    Robots.txt file is a simple text file where we gives authentication for whole website inner folders.
    Robots.txt allow to bot of every search engine.
    User agent of Robots.txt is used to do this.
  10. stacey

    stacey New Member

    Yep the robots.txt file useful for the website to protect the personal information from the public.
  11. rebeccaasmit

    rebeccaasmit New Member

    Thanks for posting this !! A well apt complete knowledge of robots.txt.....
  12. benivolentsoft

    benivolentsoft New Member

    Thanks for the detailed information and moreover Robots.txt will follow both nofollow and dofollow links

    We cant give command to the Google robots to follow the needed links because basically robot will know all the linking process for the website promotion.
  13. seo-marketing

    seo-marketing New Member

    I have seen that Google tends to ignore robots.txt file sometimes. Pages which you have specified in robots as "Disallow" are sometimes seen crawled by Google.
  14. tiwvinay

    tiwvinay New Member

    thanks for given information.
  15. seoforums85

    seoforums85 New Member

    When you do not want to crawl the page then use robot.txt file
  16. castorsandwheels

    castorsandwheels New Member

    Thanks for the information about Robot.txt.
  17. Creativepromotion

    Creativepromotion New Member

    useful info for everyone
  18. denishverma

    denishverma Denish Verma- SEO Expert

    Robots.txt is an authentication file which used to allow/ disallow folders of website.
    If disallow then anyone can not access the folder.
    If allows then can access folders, inner folders, sub folders or cgi_bin etc.

  19. jhon786

    jhon786 New Member

    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

    User-agent: *
    Disallow: /

    The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
  20. mukeshsoftona

    mukeshsoftona Banned

    Disallow or allow you website content from Google.

Share This Page