Generate robots.txt files for your website with our free tool. Control which pages search engine crawlers can and cannot access. Include your sitemap URL for better indexation.
The robots.txt file tells search engine crawlers which parts of your site they can and cannot visit. It's placed in your site's root directory and is one of the first files crawlers check.