AI Crawler Access Checker: Is Your Site Blocking GPTBot?
Paste a site and a path; it reads robots.txt and shows, for GPTBot, ClaudeBot, PerplexityBot and every other known AI crawler, whether that path is allowed and why.
- Free
- No signup
- Nothing stored
This ran once, on one input. The product runs on your whole site, every conversation.
Check what an agent answers from those pages — start freeAbout this tool
Someone asks whether the help center is blocked from ChatGPT. Someone else is sure it was opted out last year. The robots.txt has 140 lines, a wildcard block copied from a forum post, and three Allow rules that may or may not override it — and the answer depends on which user-agent you are asking about, since the file treats GPTBot, OAI-SearchBot and ChatGPT-User as three different visitors. Reading it by eye is guesswork.
Paste the site and, optionally, a path such as /help or /pricing. The tool fetches robots.txt and evaluates it the way a crawler would, for each known AI user-agent — GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent — with Googlebot and Bingbot alongside as a reference. For each it reports allowed or blocked, and the rule that decided it. Nothing you paste is stored beyond a short cache of the result.
What each input means
- 01
Site URL
The host whose robots.txt you want to test: https://www.example.com. Robots.txt is per host, so a help center on help.example.com has its own file and needs its own run. The tool fetches /robots.txt from whatever host you paste.
- 02
Path to check
Optional; defaults to /. Enter the path of a page you care about — /help, /pricing, /blog/refund-policy — and the verdict is for that path. The same file can allow the home page and block the help center, so test the page that matters.
- 03
Reading the table
One row per user-agent. Allowed means no rule forbids that path for that crawler; blocked means one does. Googlebot and Bingbot are there for comparison: if they are blocked too, the problem is the file, not an AI policy.
- 04
The deciding rule
The last column is the exact line that produced the verdict, and which user-agent group it came from. A crawler with its own group ignores the wildcard group entirely, which is the usual reason a site "blocks everything" and GPTBot still gets in.
Questions about this tool
First it picks the group: the User-agent line that best matches the crawler's name, falling back to the * group only if there is no specific one. Then, within that group, the rule with the longest matching path wins; on a tie, Allow wins. So Disallow: /help and Allow: /help/public together block /help/internal and allow /help/public/billing. The tool applies exactly that logic and shows the winning line.