Sitemaps, crawling & AI readiness
Find and check sitemaps, extract URLs, and see which AI crawlers your site lets in.
Find the Sitemap of Any Website (Sitemap Checker)
Paste a site and see every sitemap it publishes: the ones robots.txt declares and the common paths, each with its type, its URL count and whether the XML parsed.
OpenXML Sitemap Validator and Checker
Paste a sitemap URL and get a report: did it fetch, did the XML parse, index or urlset, URL count against the 50,000 limit, foreign hosts, bad dates, duplicates.
OpenSitemap URL Extractor: Extract URLs From a Sitemap
Paste a sitemap or sitemap index URL and get every page URL it lists, up to 5,000, as a plain list you can copy or download as CSV. Follows an index into its children.
OpenExtract All URLs From a Website
Paste a site and get its page URLs as a list: the crawler starts at the sitemap if there is one, otherwise the home page, and follows same-origin links up to 100 pages.
OpenRobots.txt Generator With AI Crawler Presets
Build a robots.txt in the browser: allow everything, block AI training crawlers only, block all AI crawlers or block the lot, plus your own paths, a crawl delay and the sitemap.
Openllms.txt Generator for Any Website
Paste a site and get a draft llms.txt: the site name, a one-line summary, and every page from the sitemap grouped by section with its title and description. Edit, copy, download.
OpenAI Crawler Access Checker: Is Your Site Blocking GPTBot?
Paste a site and a path; it reads robots.txt and shows, for GPTBot, ClaudeBot, PerplexityBot and every other known AI crawler, whether that path is allowed and why.
Open