[TM-Ali]

Robots.txt is a file which is use to give the instructions to the robots of search engine. We can allow and disallow the robots of search engine on a particular folder and page.

Example:-

User-agent:*
Disallow:/Folder Name/
Disallow:/page-name.html