robots.txt Generator
Tell search engines what to crawl. Allow everything, block paths, block AI training crawlers, set a crawl delay and list your sitemaps.
How to make a robots.txt
Choose what crawlers may do
Visit everything, visit everything except some paths, or visit nothing.
List paths to block or allow
One per line. Allow rules can reopen a path inside a blocked one.
Add extras
Set a crawl delay and list your sitemap URLs. You can also block AI training crawlers by name.
Save it
Download robots.txt and put it at the root of your site.
Worked example
This is what robots.txt Generator gives for its own sample values. Open the tool above and change anything to try your own.
- Crawlers may
- Visit everything except the paths below
- Block AI training crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot and more)
- No
- Crawl delay in seconds (0 for none)
- 0
robots.txt
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search Sitemap: https://example.com/sitemap.xml
About robots.txt Generator
Tell search engines which parts of your site to crawl. Allow everything, block chosen paths, block AI training crawlers by name, set a crawl delay and point to your sitemaps.
FAQs
Does robots.txt hide pages?
No. It is a request that polite crawlers follow. Pages must be protected with a login if they are private.
Where does the file go?
At the top of your site, at /robots.txt.
Does it stop AI companies using my content?
Only those that obey it. The listed bots say they do.