Robots.txt
Used to give authentication to Google bot or search engine bot for website pages and other folders.
Robots.txt file is a simple text file where we gives authentication for whole website inner folders.
Robots.txt allow to bot of every search engine.
User agent of Robots.txt is used to do this.