Go4Expert (http://www.go4expert.com/)
-   Internet Marketing (http://www.go4expert.com/articles/internet-marketing/)
-   -   Robots.txt Files (http://www.go4expert.com/articles/robotstxt-files-t16034/)

DKS 15Apr2009 21:24

Re: Robots.txt Files
Great info on robots.txt file. It is definently a tool that a lot of people don't utilise. Most likely its because they are not aware of it. Great post!

yohan 29Jul2009 06:44

Re: Robots.txt Files
It is easy for robots/spiders to crawl your site if you have this file..

chathura 3Nov2009 15:52

Re: Robots.txt Files
Good. can you show how to write when we want to say nofollow url?

technica 2Dec2009 12:30

Re: Robots.txt Files
one should also include sitemap path in the robots.txt file for better results.

devinedbyzero 26Jan2010 03:39

Re: Robots.txt Files

Originally Posted by CircuitX (Post 42350)
Ok, sorry about that :nice:.

hello bro,
follow me,you need the letter "p".you will see me in hackthis.co.uk
try to solve it.

satyedra pal 1Jun2010 11:53

Re: Robots.txt Files
We Use Robots Tag on all pages that we want indexed for the website. This will instruct the robots to crawl the page of the website.
Robot tag is implemented in following way:
<meta name="robots" content="index" />

unni krishnan.r 1Jun2010 14:06

Re: Robots.txt Files
great post

unni krishnan.r 1Jun2010 14:06

Re: Robots.txt Files
i got a vast discription

dvdv882 23Jul2010 15:20

Re: Robots.txt Files
Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.

lingoway 19Aug2010 17:05

Re: Robots.txt Files
good article

All times are GMT +5.5. The time now is 19:01.