Robots.txt Generator With AI Crawler Presets
Build a robots.txt in the browser: allow everything, block AI training crawlers only, block all AI crawlers or block the lot, plus your own paths, a crawl delay and the sitemap.
Robots.txt used to be a two-line file nobody read. Then the AI crawlers arrived — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, a dozen more — each with its own user-agent string, each doing something different with what it fetches, and someone in legal asked whether the site was "opted out". The honest answer, for most support and marketing sites, is that nobody has checked, and the file was last edited by a WordPress plugin.
This generator writes the file for you, in the browser; no request leaves the page. Pick a preset: allow everything; block AI training crawlers but leave search and answer engines in; block every known AI crawler; or block everything. Add your own disallowed paths, an optional crawl delay and your site's address for the Sitemap line, then copy the result to the root of your site. It knows thirteen AI user-agents: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and meta-externalagent.
User-agent: * Allow: /
This ran in your browser on what you typed. The product runs on your actual calls, chats and emails.
Generate an agent from the pages you leave open — start freeWhat each input means
- 01
Site URL
Your site's home page, https://www.example.com. It is used for one thing: the Sitemap: line at the end of the file, which points crawlers at /sitemap.xml on that host. Leave it blank and the line is omitted.
- 02
Preset
Allow everything is the default and what most sites should keep. Block AI training crawlers keeps search and answer engines in. Block all AI crawlers disallows all thirteen. Block everything disallows every user-agent, which is for staging sites.
- 03
Extra paths to disallow
One per line, starting with a slash: /admin, /cart, /account, /search. These are added as Disallow rules for every user-agent. Robots.txt matches prefixes, so /cart also covers /cart/checkout. Don't list anything secret here; the file is public.
- 04
Crawl delay
Seconds between requests, optional. Bing and some smaller crawlers honor it; Google ignores it and uses Search Console's rate setting instead. Leave it blank unless a crawler is measurably loading your server. Five to ten seconds is plenty.
- 05
Copy it to the site root
The file must be served at /robots.txt on each host, so www.example.com and help.example.com each need their own. Paste the output over the existing file, then run the AI crawler access checker against the live site to confirm it took.
Questions about this tool
Training crawlers — GPTBot, CCBot, Google-Extended and their kind — collect pages to train future models; blocking them costs you nothing you can measure. Answer-engine crawlers — OAI-SearchBot, PerplexityBot, ChatGPT-User — fetch a page because a user asked a question right now, and cite it, with a link. Block those and your site stops appearing in AI assistants' answers, and the referrals stop with it. The "block training, allow search" preset draws exactly that line.