> ## Documentation Index
> Fetch the complete documentation index at: https://wiredesk.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Train the agent on your website
> What the website crawl reads, its limits, what each error means, and how to read the site again.
Adding your website is the quickest way to give the agent a lot of knowledge at once. The crawler reads the public pages of your site, turns each one into text, and indexes it so the agent can answer from it on every channel.
## Add your website
Go to **Agents → your agent → Knowledge**.
Type your site's address into the box (for example `yourbusiness.com`) and click **Add website**. You can leave out `https://`, WireDesk adds it for you.
The source appears in the list as "crawling the site — this can take a minute or two…". You can leave the page, the crawl carries on without you. When it's done, the row shows how many pages and passages were indexed.
Click the source to see the pages it indexed, then try a real customer question in [Test a question](/docs/knowledge/test-a-question).
Enter the address your site actually lives at. The crawler only follows links and sitemap entries on the same host as the address you entered. If `example.com` redirects to `www.example.com`, enter `www.example.com`. A subdomain such as `help.example.com` is a separate site: add it as its own source.
## What the crawl reads
1. **The page you entered.** It's always read first and always indexed (unless you've [excluded](/docs/knowledge/keep-it-current#exclude-pages-from-a-website) it).
2. **The pages it links to.** These come next, because they're what you've chosen to put in front of visitors.
3. **Your sitemap.** The crawler reads `/sitemap.xml` at the root of your site, including up to five sub-sitemaps listed in a sitemap index.
4. **Links on those pages.** It keeps following links as it goes, so sites without a sitemap still get their deeper pages read.
The crawler reads the most useful pages first: paths that mention things like about, services, pricing, prices, menu, hours, contact, FAQ, locations or team, then shorter paths before longer ones. On a site with hundreds of blog or event pages, your pricing and contact pages still get read before the limit is reached.
A few more things to know:
* **Public pages only.** It sees what a signed-out visitor sees. Anything behind a login, a password or a cookie wall isn't read.
* **It doesn't run JavaScript.** It reads the HTML your server sends. On sites that build their pages in the browser, it still picks up each page's title and description, plus any structured business data in the page (name, address, phone, email, opening hours, price range).
* **Query strings are ignored.** `/shop?page=2` and `/shop` count as one page.
* **Files linked from your site aren't read.** Links to PDFs, images and other files are skipped. Upload documents you want the agent to know as [files](/docs/knowledge/files).
* **Pages with almost no text are skipped.** A page with 100 characters of text or fewer, apart from the page you entered, isn't indexed.
* **Web addresses only.** The address must be `http` or `https`, use a hostname rather than an IP address, and not point at a private network. Redirects are followed up to five times, and never to a private address.
The crawler identifies itself with a user agent starting `wiredesk-kb/1.0`. If your bot protection or firewall blocks unknown crawlers, allow that user agent.
## Limits
| Limit | Value |
| -------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Pages per website source | 120 |
| Text per page | 40,000 characters. Anything after that on the page is dropped. |
| Text per website source | 1,500,000 characters in total. The crawl stops when this runs out. |
| Sitemap entries considered | Up to 2,000 read, then the best 360 are queued. |
| Time per page | 20 seconds to respond. A page that takes longer is skipped. If it's the page you entered, the crawl fails. |
| Depth | No fixed depth. The crawl follows links until the page or text limit is reached, shorter paths first. |
If a large part of the site is noise the agent will never need, such as a blog archive, event listings or legal pages, [exclude those paths](/docs/knowledge/keep-it-current#exclude-pages-from-a-website). Excluded pages are never fetched, so the page and text limits go to the rest of your site.
## Errors and what they mean
The crawl only fails outright if **the page you entered** can't be read. Other pages that fail are skipped quietly, and the crawl carries on. When a crawl fails, the source shows the reason in red, with your site's hostname in place of `example.com`:
| Error | What happened | What to do |
| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| `example.com took too long to respond` | The page didn't answer within 20 seconds. | Try again later, or paste the key facts as a [note](/docs/knowledge/notes). |
| `example.com blocked the crawler` | The site refused the request (for example with a 429 or other 4xx status), or redirected somewhere the crawler won't go. | Allow the `wiredesk-kb` user agent in your bot protection, or paste the content as a note. |
| `example.com asked for a login, and we can only read public pages` | The page answered 401 or 403. | Enter a public page, or paste the content you want the agent to know. |
| `example.com has no page at that address` | The page answered 404 or 410. | Check the address, or try your home page. |
| `couldn't reach example.com` | The site returned a server error (5xx), couldn't be reached, or redirected more than five times. | Check the address and that the site is up, then add it again. |
| `example.com didn't have readable text` | The page was empty, or wasn't an HTML or plain-text page. | Enter an HTML page, or paste the content as a note. |
| `no readable text found` | The crawl finished, but no page produced any text to index. | Check the page in a private window with JavaScript off. If it's blank, paste the content as a note. |
| `this site could not be indexed after 3 attempts — it may be too large or too slow to crawl` | The crawl was interrupted three times. | Exclude large sections you don't need, or add a narrower starting page. |
| `indexing was interrupted — remove this source and add it again` | The crawl stopped and didn't recover. | Remove the source and add it again. |
You'll see one of these straight away, before anything is crawled, if the address itself isn't usable:
* `enter a website address`
* `that doesn't look like a web address`
* `the address must be http or https`
* `use a hostname, not an IP address`
* `that address is on a private network`
* `add an ending like .com to "…" — on its own it isn't a public web address`
## Re-crawling
Knowledge is a snapshot. WireDesk doesn't re-read your site on a schedule, so a page you change today stays as it was in the agent until the site is read again. There are two ways to do that:
* **Change which pages are excluded.** Open the website source, tick or untick pages, then click **Save & re-read the site**. The whole site is read again with the new exclusions. The agent keeps answering from the current pages until the new read is complete, so there's no gap. If the new read fails, the source shows the error, but the agent keeps the pages it already had.
* **Remove the source and add it again.** This reads everything fresh, but the agent has no website knowledge between removing it and the new crawl finishing.
For a single changed fact, such as new opening hours, a [note](/docs/knowledge/notes) is quicker than a re-crawl.
### Adding one page by address
Adding a specific page's address as a new website source doesn't add just that page. It starts a new crawl there: the page is read first, then the crawl carries on through the rest of the site, and you end up with a second copy of much of your site. To add one page the crawler missed, paste its content as a note instead.
## Related
* [Keep it current](/docs/knowledge/keep-it-current)
* [Notes and text](/docs/knowledge/notes)
* [Test a question](/docs/knowledge/test-a-question)
* [The agent doesn't know something](/docs/troubleshooting/agent-does-not-know)