1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots.txt Files

Discussion in 'Internet Marketing' started by CircuitX, Feb 2, 2009.

  1. PradeepKr

    PradeepKr New Member

    Joined:
    Aug 24, 2010
    Messages:
    24
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    Few cents from me,
    You can add sitemap location also in the robots.txt file.

    This would tell robots where exactly your sitemap is and which links to crawl.
     
  2. parryrater

    parryrater New Member

    Joined:
    Aug 12, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    hi...

    really robot.txt file has more useful to make user friendly site..!!! meet again.
     
  3. raantint

    raantint New Member

    Joined:
    Aug 11, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    hi...

    Is there any robot.txt generator tool like as sitemap generator...??? meet again.
     
  4. shabbir

    shabbir Administrator Staff Member

    Joined:
    Jul 12, 2004
    Messages:
    15,293
    Likes Received:
    365
    Trophy Points:
    83
    Visit Google Webmaster tool and they have option for generating one for your domain
     
  5. parrytint

    parrytint New Member

    Joined:
    Aug 9, 2010
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    hi...

    What are the benefits by the robots.txt file generate..??? meet again.
     
  6. raanzen

    raanzen New Member

    Joined:
    Aug 10, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    hi...

    Which URL's may not eligible to crawl by search engine in any any site...??? meet again.
     
  7. vimlesh

    vimlesh Banned

    Joined:
    Feb 24, 2011
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    0
    The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.
     
  8. delhifirm

    delhifirm New Member

    Joined:
    Mar 3, 2011
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    hello
    Thank for sharing.
    It is a basic notes on Robots.txt Files that is good
     
  9. denishverma

    denishverma Denish Verma- SEO Expert

    Joined:
    Aug 7, 2010
    Messages:
    124
    Likes Received:
    15
    Trophy Points:
    0
    Occupation:
    Sr. Web Developer
    Location:
    INDIA
    Home Page:
    Robots.txt
    Used to give authentication to Google bot or search engine bot for website pages and other folders.
    Robots.txt file is a simple text file where we gives authentication for whole website inner folders.
    Robots.txt allow to bot of every search engine.
    User agent of Robots.txt is used to do this.
     
  10. stacey

    stacey New Member

    Joined:
    Jan 31, 2011
    Messages:
    60
    Likes Received:
    1
    Trophy Points:
    0
    Occupation:
    Professional
    Location:
    Nashua, NH
    Home Page:
    Yep the robots.txt file useful for the website to protect the personal information from the public.
     
  11. rebeccaasmit

    rebeccaasmit New Member

    Joined:
    Jun 7, 2011
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    0
    Thanks for posting this !! A well apt complete knowledge of robots.txt.....
     
  12. benivolentsoft

    benivolentsoft New Member

    Joined:
    Feb 2, 2011
    Messages:
    21
    Likes Received:
    0
    Trophy Points:
    0
    Home Page:
    Thanks for the detailed information and moreover Robots.txt will follow both nofollow and dofollow links

    We cant give command to the Google robots to follow the needed links because basically robot will know all the linking process for the website promotion.
     
  13. seo-marketing

    seo-marketing New Member

    Joined:
    Jun 29, 2009
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    Occupation:
    Website Promotion SEO and Marketing Executive havi
    Location:
    India
    Home Page:
    I have seen that Google tends to ignore robots.txt file sometimes. Pages which you have specified in robots as "Disallow" are sometimes seen crawled by Google.
     
  14. tiwvinay

    tiwvinay New Member

    Joined:
    May 7, 2011
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    0
    thanks for given information.
     
  15. seoforums85

    seoforums85 New Member

    Joined:
    Jul 6, 2011
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    0
    When you do not want to crawl the page then use robot.txt file
     
  16. castorsandwheels

    castorsandwheels New Member

    Joined:
    Aug 4, 2011
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    United Kingdom
    Home Page:
    Thanks for the information about Robot.txt.
     
  17. Creativepromotion

    Creativepromotion New Member

    Joined:
    Dec 11, 2010
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    0
    Occupation:
    SEO
    Location:
    Mumbai
    Home Page:
    useful info for everyone
     
  18. denishverma

    denishverma Denish Verma- SEO Expert

    Joined:
    Aug 7, 2010
    Messages:
    124
    Likes Received:
    15
    Trophy Points:
    0
    Occupation:
    Sr. Web Developer
    Location:
    INDIA
    Home Page:
    Robots.txt is an authentication file which used to allow/ disallow folders of website.
    If disallow then anyone can not access the folder.
    If allows then can access folders, inner folders, sub folders or cgi_bin etc.

    thanks
     
  19. jhon786

    jhon786 New Member

    Joined:
    Oct 12, 2011
    Messages:
    46
    Likes Received:
    0
    Trophy Points:
    0
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

    User-agent: *
    Disallow: /

    The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
     
  20. mukeshsoftona

    mukeshsoftona Banned

    Joined:
    Oct 28, 2011
    Messages:
    47
    Likes Received:
    0
    Trophy Points:
    0
    Disallow or allow you website content from Google.
     

Share This Page