DKS's Avatar
DKS
Light Poster
Great info on robots.txt file. It is definently a tool that a lot of people don't utilise. Most likely its because they are not aware of it. Great post!
yohan's Avatar, Join Date: Jun 2009
Ambitious contributor
It is easy for robots/spiders to crawl your site if you have this file..
chathura's Avatar, Join Date: Oct 2009
Go4Expert Member
Good. can you show how to write when we want to say nofollow url?
technica's Avatar, Join Date: Dec 2007
CoderByNature
one should also include sitemap path in the robots.txt file for better results.
devinedbyzero's Avatar, Join Date: Jan 2010
Newbie Member
Quote:
Originally Posted by CircuitX View Post
Ok, sorry about that .
hello bro,
follow me,you need the letter "p".you will see me in hackthis.co.uk
try to solve it.
satyedra pal's Avatar, Join Date: Mar 2010
Contributor
We Use Robots Tag on all pages that we want indexed for the website. This will instruct the robots to crawl the page of the website.
Robot tag is implemented in following way:
<meta name="robots" content="index" />
unni krishnan.r's Avatar, Join Date: Apr 2010
Skilled contributor
great post
unni krishnan.r's Avatar, Join Date: Apr 2010
Skilled contributor
i got a vast discription
dvdv882's Avatar, Join Date: Jul 2010
Banned
Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.
lingoway's Avatar
Go4Expert Member
good article