Extract All URLs From a Website
Paste a site and get its page URLs as a list: the crawler starts at the sitemap if there is one, otherwise the home page, and follows same-origin links up to 100 pages.
Not every site has a sitemap, and not every sitemap is honest. The marketing site has been through three agencies; the help center is on a subdomain nobody registered in robots.txt; the pricing page links to a FAQ that exists nowhere else. When you want to know what a crawler will actually find — before you point an agent at it, or a vendor does — the sitemap is a claim and a crawl is the evidence.
Paste the site's address. The tool reads the sitemap first if there is one, then crawls from the home page, breadth-first, following links on the same origin. It stops at 100 pages or about 20 seconds, whichever comes first, and returns the URLs it reached as a list you can copy or download as CSV. The cap is deliberate: this is a sample of what a crawler sees, not a full inventory. The crawl uses WireDesk's own crawler, the one that reads a customer's site to build an agent. Nothing you paste is stored beyond a short cache of the result.
This ran once, on one input. The product runs on your whole site, every conversation.
Crawl your site and put an agent on it — start freeWhat each input means
- 01
Site URL
The home page, with the scheme: https://www.example.com. The crawl stays on that origin, so help.example.com and shop.example.com are separate runs. Paste a deeper URL such as /help to start the crawl from there instead.
- 02
How the crawl runs
Sitemap first, if robots.txt or a common path reveals one. Then the home page, then every same-origin link on it, then every link on those pages — breadth-first, so the list fills with the shallowest pages before it goes deep.
- 03
The caps
100 pages, or about 20 seconds, whichever comes first. A slow site hits the time cap first and returns fewer URLs. If the count reads 100, there is more; the list shows what a crawler reaches first, which is usually what matters.
- 04
Copy or download CSV
Copy puts the list on the clipboard, one URL per line. Download CSV saves the same list as a file. Compare it against the sitemap's list to see which pages are linked but undeclared, or declared but unreachable.
Questions about this tool
Because a free crawl that ran for minutes against someone else's server would be a nuisance, and because 100 breadth-first pages already show you what a crawler finds first: the navigation, the top-level sections, the first layer of the help center. For a full inventory, extract the sitemap instead, up to 5,000 URLs at a time.