AI Crawler Access Checker: Is Your Site Blocking GPTBot?
Paste a site and a path; it reads robots.txt and shows, for GPTBot, ClaudeBot, PerplexityBot and every other known AI crawler, whether that path is allowed and why.
Someone asks whether the help center is blocked from ChatGPT. Someone else is sure it was opted out last year. The robots.txt has 140 lines, a wildcard block copied from a forum post, and three Allow rules that may or may not override it — and the answer depends on which user-agent you are asking about, since the file treats GPTBot, OAI-SearchBot and ChatGPT-User as three different visitors. Reading it by eye is guesswork.
Paste the site and, optionally, a path such as /help or /pricing. The tool fetches robots.txt with WireDesk's crawler and evaluates it the way a crawler would, for each known AI user-agent — GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent — with Googlebot and Bingbot alongside as a reference. For each it reports allowed or blocked, and the rule that decided it. Nothing you paste is stored beyond a short cache of the result.
This ran once, on one input. The product runs on your whole site, every conversation.
Check what an agent answers from those pages — start freeWhat each input means
- 01
Site URL
The host whose robots.txt you want to test: https://www.example.com. Robots.txt is per host, so a help center on help.example.com has its own file and needs its own run. The tool fetches /robots.txt from whatever host you paste.
- 02
Path to check
Optional; defaults to /. Enter the path of a page you care about — /help, /pricing, /blog/refund-policy — and the verdict is for that path. The same file can allow the home page and block the help center, so test the page that matters.
- 03
Reading the table
One row per user-agent. Allowed means no rule forbids that path for that crawler; blocked means one does. Googlebot and Bingbot are there for comparison: if they are blocked too, the problem is the file, not an AI policy.
- 04
The deciding rule
The last column is the exact line that produced the verdict, and which user-agent group it came from. A crawler with its own group ignores the wildcard group entirely, which is the usual reason a site "blocks everything" and GPTBot still gets in.
Questions about this tool
First it picks the group: the User-agent line that best matches the crawler's name, falling back to the * group only if there is no specific one. Then, within that group, the rule with the longest matching path wins; on a tie, Allow wins. So Disallow: /help and Allow: /help/public together block /help/internal and allow /help/public/billing. The tool applies exactly that logic and shows the winning line.