robots.txt Generator

Tell search engines what to crawl. Allow everything, block paths, block AI training crawlers, set a crawl delay and list your sitemaps.

Processed on your device. Files are not uploaded

How to make a robots.txt

  1. Choose what crawlers may do

    Visit everything, visit everything except some paths, or visit nothing.

  2. List paths to block or allow

    One per line. Allow rules can reopen a path inside a blocked one.

  3. Add extras

    Set a crawl delay and list your sitemap URLs. You can also block AI training crawlers by name.

  4. Save it

    Download robots.txt and put it at the root of your site.

Worked example

This is what robots.txt Generator gives for its own sample values. Open the tool above and change anything to try your own.

Crawlers may
Visit everything except the paths below
Block AI training crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot and more)
No
Crawl delay in seconds (0 for none)
0

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

About robots.txt Generator

Tell search engines which parts of your site to crawl. Allow everything, block chosen paths, block AI training crawlers by name, set a crawl delay and point to your sitemaps.

FAQs

Does robots.txt hide pages?

No. It is a request that polite crawlers follow. Pages must be protected with a login if they are private.

Where does the file go?

At the top of your site, at /robots.txt.

Does it stop AI companies using my content?

Only those that obey it. The listed bots say they do.

Related tools