Skip to main content
Adding your website is the quickest way to give the agent a lot of knowledge at once. The crawler reads the public pages of your site, turns each one into text, and indexes it so the agent can answer from it on every channel.

​
Add your website

1

Open Knowledge

Go to Agents → your agent → Knowledge.
2

Enter the address

Type your site’s address into the box (for example yourbusiness.com) and click Add website. You can leave out https://, WireDesk adds it for you.
3

Wait for it to finish

The source appears in the list as “crawling the site — this can take a minute or two…”. You can leave the page, the crawl carries on without you. When it’s done, the row shows how many pages and passages were indexed.
4

Check what it read

Click the source to see the pages it indexed, then try a real customer question in Test a question.
Enter the address your site actually lives at. The crawler only follows links and sitemap entries on the same host as the address you entered. If example.com redirects to www.example.com, enter www.example.com. A subdomain such as help.example.com is a separate site: add it as its own source.

​
What the crawl reads

  1. The page you entered. It’s always read first and always indexed (unless you’ve excluded it).
  2. The pages it links to. These come next, because they’re what you’ve chosen to put in front of visitors.
  3. Your sitemap. The crawler reads /sitemap.xml at the root of your site, including up to five sub-sitemaps listed in a sitemap index.
  4. Links on those pages. It keeps following links as it goes, so sites without a sitemap still get their deeper pages read.
The crawler reads the most useful pages first: paths that mention things like about, services, pricing, prices, menu, hours, contact, FAQ, locations or team, then shorter paths before longer ones. On a site with hundreds of blog or event pages, your pricing and contact pages still get read before the limit is reached. A few more things to know:
  • Public pages only. It sees what a signed-out visitor sees. Anything behind a login, a password or a cookie wall isn’t read.
  • It doesn’t run JavaScript. It reads the HTML your server sends. On sites that build their pages in the browser, it still picks up each page’s title and description, plus any structured business data in the page (name, address, phone, email, opening hours, price range).
  • Query strings are ignored. /shop?page=2 and /shop count as one page.
  • Files linked from your site aren’t read. Links to PDFs, images and other files are skipped. Upload documents you want the agent to know as files.
  • Pages with almost no text are skipped. A page with 100 characters of text or fewer, apart from the page you entered, isn’t indexed.
  • Web addresses only. The address must be http or https, use a hostname rather than an IP address, and not point at a private network. Redirects are followed up to five times, and never to a private address.
The crawler identifies itself with a user agent starting wiredesk-kb/1.0. If your bot protection or firewall blocks unknown crawlers, allow that user agent.

​
Limits

If a large part of the site is noise the agent will never need, such as a blog archive, event listings or legal pages, exclude those paths. Excluded pages are never fetched, so the page and text limits go to the rest of your site.

​
Errors and what they mean

The crawl only fails outright if the page you entered can’t be read. Other pages that fail are skipped quietly, and the crawl carries on. When a crawl fails, the source shows the reason in red, with your site’s hostname in place of example.com: You’ll see one of these straight away, before anything is crawled, if the address itself isn’t usable:
  • enter a website address
  • that doesn't look like a web address
  • the address must be http or https
  • use a hostname, not an IP address
  • that address is on a private network
  • add an ending like .com to "…" — on its own it isn't a public web address

​
Re-crawling

Knowledge is a snapshot. WireDesk doesn’t re-read your site on a schedule, so a page you change today stays as it was in the agent until the site is read again. There are two ways to do that:
  • Change which pages are excluded. Open the website source, tick or untick pages, then click Save & re-read the site. The whole site is read again with the new exclusions. The agent keeps answering from the current pages until the new read is complete, so there’s no gap. If the new read fails, the source shows the error, but the agent keeps the pages it already had.
  • Remove the source and add it again. This reads everything fresh, but the agent has no website knowledge between removing it and the new crawl finishing.
For a single changed fact, such as new opening hours, a note is quicker than a re-crawl.

​
Adding one page by address

Adding a specific page’s address as a new website source doesn’t add just that page. It starts a new crawl there: the page is read first, then the crawl carries on through the rest of the site, and you end up with a second copy of much of your site. To add one page the crawler missed, paste its content as a note instead.