Find All URLs on a Website: Free Page URL Extractor
Paste a site and get its page URLs as a list: the crawler starts at the sitemap if there is one, otherwise the home page, and follows same-origin links up to 100 pages.
- Free
- No signup
- Nothing stored
This ran once, on one input. The product runs on your whole site, every conversation.
Crawl your site and put an agent on it — start freeAbout this tool
Not every site has a sitemap, and not every sitemap is honest. The marketing site has been through three agencies; the help center is on a subdomain nobody registered in robots.txt; the pricing page links to a FAQ that exists nowhere else. When you want to know what a crawler will actually find — before you point an agent at it, or a vendor does — the sitemap is a claim and a crawl is the evidence.
Paste the site's address. The tool reads the sitemap first if there is one, then crawls from the home page, breadth-first, following links on the same origin. It stops at 100 pages or about 20 seconds, whichever comes first, and returns the URLs it reached as a list you can copy or download as CSV. The cap is deliberate: this is a sample of what a crawler sees, not a full inventory. Requests identify themselves as wiredesk-tools in the site's logs. Nothing you paste is stored beyond a short cache of the result.
How to get all the page URLs from a website: three ways
1. Read the sitemap. Most sites publish one at /sitemap.xml, or name it on a Sitemap: line in /robots.txt. It lists the pages the owner wants found, often with a last-modified date, but it can be missing, stale or split into several files. 2. Crawl the links. Start at the home page and follow every link on the same site, layer by layer. This finds what a visitor or a search engine can actually reach, including pages the sitemap forgot. It is what this tool does, up to 100 pages. 3. Ask the search engine. For a site you own, Search Console's Pages report lists what Google has indexed. For anyone's site, a site:example.com search shows a sample, though it undercounts. Compare the first two: a page in the sitemap that no link reaches is orphaned, and a linked page missing from the sitemap is undeclared.
What each input means
- 01
Site URL
The home page, with the scheme: https://www.example.com. The crawl stays on that origin, so help.example.com and shop.example.com are separate runs. Paste a deeper URL such as /help to start the crawl from there instead.
- 02
How the crawl runs
Sitemap first, if robots.txt or a common path reveals one. Then the home page, then every same-origin link on it, then every link on those pages — breadth-first, so the list fills with the shallowest pages before it goes deep.
- 03
The caps
100 pages, or about 20 seconds, whichever comes first. A slow site hits the time cap first and returns fewer URLs. If the count reads 100, there is more; the list shows what a crawler reaches first, which is usually what matters.
- 04
Copy or download CSV
Copy puts the list on the clipboard, one URL per line. Download CSV saves the same list as a file. Compare it against the sitemap's list to see which pages are linked but undeclared, or declared but unreachable.
Questions about this tool
Start with the sitemap, usually at /sitemap.xml or named in /robots.txt, which lists the pages the owner declared. Then crawl the links from the home page to find what is actually reachable, which is what this tool does: paste the site and it reads the sitemap, follows same-site links breadth-first, and returns up to 100 URLs to copy or download as CSV. For a site you own, Search Console lists what Google has indexed.