Skip to main content
Adding your website is the quickest way to give the agent a lot of knowledge at once. The crawler reads the public pages of your site, turns each one into text, and indexes it so the agent can answer from it on every channel.

​
Add your website

1

Open Knowledge

Go to Agents → your agent → Knowledge.
2

Enter the address

Click the Website tile, type your site’s address (for example yourcompany.com) and click Find pages. You can leave out https://, Wiredesk adds it for you.
3

Pick the pages

Wiredesk maps the site from its sitemap and home page in a few seconds, without reading anything yet, and shows it as a tree of sections with a page count beside each. Everything is ticked. Untick the sections customers never ask about, such as a blog, a forum, job ads or a changelog, so your knowledge allowance goes to the pages they do ask about. The card at the top estimates how many of the picked pages fit in what your plan has left. Then click Read pages.
4

Wait for it to finish

The source appears in the list as “Reading the site…”, then counts the pages as they’re read. A small site takes a minute or two; a site of thousands of pages can take a few hours. You can leave the page, the crawl carries on without you, and the agent can answer from pages as soon as they’re read. When it’s done, the row shows how many pages and passages were indexed.
5

Check what it read

Click the source to see the pages it indexed, then try a real customer question in Test a question.
Enter the address your site actually lives at. The crawler only follows links and sitemap entries on the same host as the address you entered. If example.com redirects to www.example.com, enter www.example.com. A subdomain such as help.example.com is a separate site: add it as its own source.

​
What the crawl reads

  1. The page you entered. It’s always read first and always indexed (unless you’ve excluded it).
  2. Your key pages and navigation. Top-level pages named for what customers ask about, such as /pricing, /about, /contact, /faq, /services, /hours or /menu, then the other top-level pages your home page links to. Together these take at most a fifth of what your allowance is expected to read.
  3. Every section, in proportion to its size. Everything else from your sitemap and home page is shared between your site’s sections by how many pages each has, shallow pages first within each. If /docs is 70% of what you ticked, it gets about 70% of the pages read, and a small section is never starved by a big one.
  4. Links on those pages. It keeps following links as it goes, so sites without a sitemap still get their deeper pages read.
The crawler reads /sitemap.xml at the root of your site, including up to five sub-sitemaps listed in a sitemap index. Link lists and feeds such as llms.txt or an RSS feed aren’t read as pages. A few more things to know:
  • Public pages only. It sees what a signed-out visitor sees. Anything behind a login, a password or a cookie wall isn’t read.
  • Sites built in the browser are read in a browser. The crawler first reads the HTML your server sends, which is fast and covers most sites. If the page you entered comes back nearly empty, as an app built with React, Vue or similar does before its JavaScript runs, the crawler loads it in a real browser and reads what it draws. If that worked, the rest of the site’s nearly empty pages are read the same way, up to 40 pages per crawl.
  • Query strings are ignored. /shop?page=2 and /shop count as one page.
  • Files linked from your site aren’t read. Links to PDFs, images and other files are skipped. Upload documents you want the agent to know as files.
  • Pages with almost no text are skipped. A page with 100 characters of text or fewer, apart from the page you entered, isn’t indexed.
  • Web addresses only. The address must be http or https, use a hostname rather than an IP address, and not point at a private network. Redirects are followed up to five times, and never to a private address.
The crawler identifies itself with a user agent starting wiredesk-kb/1.0. If your bot protection or firewall blocks unknown crawlers, allow that user agent. When a page answers 403, the crawler tries it once more in a browser (Cloudflare Browser Run), which also identifies itself as a bot. It doesn’t solve CAPTCHAs or challenges, so a site that refuses crawlers refuses it too.

​
Limits

When the allowance runs out before the site does, the crawl stops there and the source is ready with a note: “Stopped at your plan’s 5 MB of knowledge — the most useful pages were read first.” If a large part of the site is noise the agent will never need, such as a blog archive, event listings or legal pages, exclude those paths. Excluded pages are never fetched, so the allowance goes to the rest of your site.

​
Errors and what they mean

The crawl only fails outright if the page you entered can’t be read. Other pages that fail are skipped quietly, and the crawl carries on. When a crawl fails, the source shows the reason in red, with your site’s hostname in place of example.com: You’ll see one of these straight away, before anything is crawled, if the address itself isn’t usable:
  • enter a website address
  • that doesn't look like a web address
  • the address must be http or https
  • use a hostname, not an IP address
  • that address is on a private network
  • add an ending like .com to "…" — on its own it isn't a public web address

​
Re-crawling

Knowledge is a snapshot. Wiredesk doesn’t re-read your site on a schedule, so a page you change today stays as it was in the agent until the site is read again. There are two ways to do that:
  • Change which pages are excluded. Open the website source, tick or untick pages, then click Save & re-read the site. The whole site is read again with the new exclusions. The agent keeps answering from the current pages until the new read is complete, so there’s no gap. If the new read fails, the source shows the error, but the agent keeps the pages it already had.
  • Remove the source and add it again. This reads everything fresh, but the agent has no website knowledge between removing it and the new crawl finishing.
For a single changed fact, such as new opening hours, a note is quicker than a re-crawl.

​
Adding one page by address

Adding a specific page’s address as a new website source doesn’t add just that page. It starts a new crawl there: the page is read first, then the crawl carries on through the rest of the site, and you end up with a second copy of much of your site. To add one page the crawler missed, paste its content as a note instead.