Robots.txt is important from an SEO point of view. By specifying where SE bots should look for content, you can increase the search engine ranking of your site.
From an SEO perspective, robots.txt is an important file that contains instructions to search engine spiders that tells them which pages are to be indexed and made public, and which pages should not be crawled.
