Knowledge sources
Teach your agent about your business by adding knowledge from websites, files, text, or Q&A pairs.
Open your agent's Knowledge page →
Four types of knowledge
Your agent can learn from four different source types. You can mix and match as many as you need, up to your plan's memory limit.
Website URLs
Provide a URL and Dante AI will crawl the page and follow internal links, extracting visible text from each page it finds. This is the fastest way to get started if your business already has a website with relevant information.
- Enter the full URL including
https:// - The crawler stays on the same domain. Subdomains and external sites need their own source.
- Add as many URLs as your memory limit allows
Crawl-time URL filters automatically skip low-value paths
(/blog, /news, /legal,
/privacy, /terms, /login,
/tag, /search and similar) so generic
boilerplate doesn't dilute your product chunks. If your product
actually lives under /blog, point the seed
URL at example.com/blog and we'll crawl /blog/*
normally.
When you first set up your agent, we read up to 25 pages of the site you enter to train it, so your very first answers are grounded in your real content. Crawling only reads your public pages; your website is never changed. You can add more URLs (up to your memory limit) any time from the Knowledge screen for a deeper knowledge base.
File uploads
Upload documents directly. This works well for product manuals, policy documents, and internal guides that aren't published online.
- Documents:
.pdf,.docx,.doc,.txt,.md,.rtf,.epub - Spreadsheets:
.xlsx,.csv - Slides:
.pptx - Structured:
.json,.xml,.html - Images:
.jpg,.png(text is extracted via OCR)
Upload limits and how big batches work
- Per file: up to 50 MB. Files over that are rejected before upload with a clear message; split the file into smaller parts and add each one.
- Big batches: drop as many files as you like at once. Uploads are sent in parts automatically (up to 25 files per part), with live per-file progress.
- If a part fails: files that already uploaded are kept and start training; only the failed files are listed, each with the reason, and a single button retries them. A failed upload is a technical problem on our side, not a plan limit, so you never need to upgrade to fix one.
Text input
Type or paste text directly into the knowledge editor. Use this for custom content that doesn't exist anywhere else. Internal policies, specific talking points, brand-voice guidelines, or product details you want your agent to know.
Q&A pairs
Add specific question-and-answer pairs to guarantee your agent gives exact responses for common queries. When a visitor asks a question that closely matches one of your Q&A pairs, the agent uses your provided answer.
- Great for FAQs, return policies, pricing questions
- Ensures precise, consistent answers
- Takes priority over scraped website content for matching questions
Memory limits by plan
Each plan has a maximum amount of knowledge your agent can store, measured in characters. Every letter, space, and punctuation mark counts toward your agent's knowledge cap.
For context: a typical web page is about 2,000 characters, a short PDF around 5,000, and a long blog article around 15,000.
| Plan | Memory limit | Roughly equivalent to |
|---|---|---|
| Free | 150,000 characters | ~75 web pages |
| Starter | 25,000,000 characters | ~12,500 web pages |
| Advanced | 50,000,000 characters | ~25,000 web pages |
| Pro | 200,000,000 characters | ~100,000 web pages |
Managing your knowledge
Adding sources
- Open Knowledge. In your agent sidebar, click Knowledge.
- Choose a source type. Select from Website, Files, Text, or Q&A at the top of the panel.
- Add your content. Enter a URL, upload a file, type text, or create a Q&A pair.
- Save. Your agent immediately learns from the new source. No separate training step required.
Editing sources
Click on any existing knowledge source to edit its content. Changes take effect immediately. For website URLs, you can re-crawl the page to pull the latest content.
Deleting sources
Click the delete icon next to any source to remove it. Your agent will no longer reference that content when answering questions. Deleting a source frees up memory toward your plan limit.
How the agent uses your knowledge
On every reply, your agent embeds the visitor's message, retrieves the top-matching chunks from your knowledge index, and uses those chunks as the source of truth for the answer. It does not reach into general internet knowledge to guess about your product. If the chunks don't cover the question, it says so honestly rather than making something up.
Auto-update knowledge
Advanced and Pro plans can re-crawl website sources on a schedule, so your agent stays current without manual re-training. Pick a cadence in the Training summary sidebar under Auto-update.
| Plan | Auto-update cadence |
|---|---|
| Free | Not available |
| Starter | Not available |
| Advanced | Weekly or Monthly |
| Pro | Daily, Every 3 days, or Weekly |
Troubleshooting
Training is stuck, or a crawl came back empty
Symptom. A website source sits on "Training" and never finishes, or it completes but adds little or no text and the agent says it doesn't know things that are clearly on the page.
Cause. The crawler reads the visible text a page returns. It comes back thin when the content is rendered entirely by JavaScript after load, sits behind a login or cookie wall, is mostly images or PDFs with no underlying text, or the seed URL points at a page whose real content lives on other paths. A crawl can also look "stuck" while it is still working through a large site.
What you'll see. As the crawl runs, the page count climbs live so you can see it working rather than watching a silent spinner. When a crawl finishes with almost no readable text, the source is marked Needs attention instead of appearing as a successful crawl. The Website page shows a warning ("We could not read content from this site") with a one-click Retry crawl button and shortcuts to add the content another way (another URL, a file upload, or pasted text). During your first crawl, if it comes back with barely a page, you'll see a recovery prompt right there with one-click options to add a specific page, upload a document, paste your text, or try the AI agent anyway, so you are never quietly handed an agent that has nothing to answer from. The Publish page also warns you before you go live if your AI agent's knowledge base is still nearly empty. If the crawl did capture a little text, the sandbox answers from that content rather than telling you it has no knowledge, so what you test is what your visitors would get.
Fix. Give it a minute on large sites, the status updates as pages stream in. If it finishes thin, use Retry crawl on the warning, point the seed URL straight at the page that holds the content (for example example.com/docs rather than the homepage), or add that content directly as a file upload, text, or Q&A pair so it does not depend on the crawler. When a site publishes its page list in a sitemap that we did not reach on the first pass, a retry can pick up those pages automatically. Remember the crawler skips low-value paths like /blog and /legal by default, so seed those explicitly if your product lives there.
The site blocked our crawler, so only the first page trained
Symptom. A website source is marked Needs attention with the message "This site blocked our AI agent's crawler, so we could only read its first page", even though that first page has plenty of content.
Cause. Some sites refuse automated readers (a bot-protection service, or a rule that blocks non-browser requests). When that happens, the crawler cannot follow links past the homepage, so it reads the seed page alone rather than the whole site. We now flag this instead of showing a successful crawl, because your AI agent would otherwise answer from a single page while looking fully trained.
Fix. Point the seed URL at a section that allows crawling (a docs, help, or landing subdomain often works), or add the rest of your content directly as file uploads, text, or Q&A pairs so it does not depend on the crawler. If you control the site's bot-protection settings, allowing verified crawlers lets a Retry crawl read the full site next time.
The site's security layer blocked the crawl completely
Symptom. A website source is marked Needs attention with the message "This site's security layer blocked our crawler. Try another site, or upload a document instead." There is no Retry crawl button on this warning.
Cause. Some sites sit behind a security service that shows every automated visitor a verification page (often titled "Just a moment...") instead of the real content. When that happens, even the first page the crawler reads is the verification screen, not your site, so there is genuinely nothing for your AI agent to learn from. We stop here on purpose: retrying the same address hits the same wall, and earlier versions of this flow could quietly train an agent on the verification page itself, which left it unable to answer anything about the business.
Fix. Add your content a way that does not depend on the crawler: upload a PDF or document, paste text, or add Q&A pairs, the agent learns from those just as well as from pages. A different address on the same business can also work when it is served outside the security layer (a docs, help, or landing subdomain often is). If you control the site's security settings, allowing verified crawlers lets a future crawl of the site succeed.
Every crawled page came back identical (the site redirects to one page)
Symptom. A website source is marked Needs attention with the message "Most crawled pages returned identical content", or an earlier crawl shows many pages that all carry the same title and the same content while your AI agent only ever answers from the homepage.
Cause. Some sites redirect every address to a single page, most commonly a redirect from the bare domain to the www address that drops the rest of the link. Each page the crawler asked for bounced to the same place, so instead of your real pages it read one page many times. The crawler now follows each redirect to its final address, keeps one copy of each page, and skips the duplicates, and when a finished crawl still comes back mostly identical or with most pages unreadable it is flagged Needs attention instead of appearing as a successful crawl.
Fix. Crawl the site's full address, including www if that is where your site actually lives (for example https://www.example.com rather than https://example.com), then use Retry crawl. If the flag persists, point the seed URL straight at the section that holds your content, or add it directly as a file upload, text, or Q&A pairs. If it still does not look right, contact support, this flag also alerts our team.
Some pages were skipped as "not found"
Symptom. After a crawl, the Website page shows a note like "3 pages skipped: not found", and those rows carry a Skipped badge with the detail "This page shows a not found message instead of real content".
Cause. Your site linked to addresses that no longer exist, but instead of returning a proper error, they show a "page not found" message styled like a normal page. Left alone, those error pages would train into your AI agent's knowledge and could surface in its answers. Dante detects them at training time and leaves them out; they use none of your storage. Real pages that merely talk about errors (say, an article about handling 404s) are not affected, the check only skips pages whose title or content is the error message itself.
Fix. Usually nothing, the skip is doing its job and your agent trains on the real pages. If a skipped address should exist, fix the page or the link on your site, then select the row and click Train (or use Retry crawl) and it will be read again.
A file upload failed, or only some files went through
Symptom. After dropping a batch of files, some rows show a red error while others uploaded fine.
Cause. Either a file is over the 50 MB per-file limit, or the upload service had a temporary problem sending one of the parts. Uploads are sent in parts automatically, so one failed part never takes down the rest of the batch.
Fix. Files that uploaded are already saved and training; you do not need to re-add them. For a file over 50 MB, split it into smaller files and add those. For anything else, use the Retry failed files button on the upload panel. This kind of failure is technical, not a plan limit, so upgrading your plan will not change it; if retries keep failing, contact support.
"We couldn't read your site" during onboarding
Symptom. While setting up your first AI agent, the add-your-site step reports that your site couldn't be read, and offers to try a different URL, upload a document, or continue anyway.
Cause. Some sites protect themselves from automated readers (for example with a bot-protection or challenge page). The crawler receives that protection page instead of your real content, so there is nothing for your AI agent to learn from.
Fix. Try a URL that is not behind the protection (a docs, help, or landing subdomain often works), or upload a PDF or document with the same content instead, the agent learns from files just as well as from pages. You can also continue and add knowledge later from the dashboard under Knowledge. If your site uses a bot-protection service you control, allowing verified crawlers in its settings lets future re-crawls succeed.
One exception. When the step says "This site's security layer blocked our crawler", the crawl was blocked completely and there is no continue-anyway option: an agent trained at that point would have nothing at all to answer from. Pick another site or upload a document instead, both take seconds and get you to a working AI agent.
Still stuck? We'll help.
Every plan, including Free, comes with human support. If something isn't behaving the way you expect after following the steps above, send us a note and we'll get back to you within one business day.