It helps crawler for crawler because all urls which we want to crawl is indexed in a page.