Sitemaps, crawling & AI readiness
Find and check sitemaps, extract URLs, and see which AI crawlers your site lets in. Free, no signup.
Find the Sitemap of Any Website (Sitemap Checker)Paste a site and see every sitemap it publishes: the ones robots.txt declares and the common paths, each with its type, its URL count and whether the XML parsed.XML Sitemap Validator and CheckerPaste a sitemap URL and get a report: did it fetch, did the XML parse, index or urlset, URL count against the 50,000 limit, foreign hosts, bad dates, duplicates.Sitemap URL Extractor: Extract URLs From a SitemapPaste a sitemap or sitemap index URL and get every page URL it lists, up to 5,000, as a plain list you can copy or download as CSV. Follows an index into its children.Find All URLs on a Website: Free Page URL ExtractorPaste a site and get its page URLs as a list: the crawler starts at the sitemap if there is one, otherwise the home page, and follows same-origin links up to 100 pages.Robots.txt Generator With AI Crawler PresetsBuild a robots.txt in the browser: allow everything, block AI training crawlers only, block all AI crawlers or block the lot, plus your own paths, a crawl delay and the sitemap.llms.txt Generator for Any WebsitePaste a site and get a draft llms.txt: the site name, a one-line summary, and every page from the sitemap grouped by section with its title and description. Edit, copy, download.AI Crawler Access Checker: Is Your Site Blocking GPTBot?Paste a site and a path; it reads robots.txt and shows, for GPTBot, ClaudeBot, PerplexityBot and every other known AI crawler, whether that path is allowed and why.