Knowledge sources
Teach your agent about your business by adding knowledge from websites, files, text, or Q&A pairs.
Open your agent's Knowledge page →
Four types of knowledge
Your agent can learn from four different source types. You can mix and match as many as you need, up to your plan's memory limit.
Website URLs
Provide a URL and Dante AI will crawl the page and follow internal links, extracting visible text from each page it finds. This is the fastest way to get started if your business already has a website with relevant information.
- Enter the full URL including
https:// - The crawler stays on the same domain. Subdomains and external sites need their own source.
- Add as many URLs as your memory limit allows
Crawl-time URL filters automatically skip low-value paths
(/blog, /news, /legal,
/privacy, /terms, /login,
/tag, /search and similar) so generic
boilerplate doesn't dilute your product chunks. If your product
actually lives under /blog, point the seed
URL at example.com/blog and we'll crawl /blog/*
normally.
When you first set up your agent, that first crawl is deliberately bounded: we read a capped set of the site's core pages rather than the whole site, so your very first answers are grounded in your real content. The exact page cap depends on how your agent was created and can change as we tune the first run, so the Website tab is the source of truth: once the crawl finishes it lists every page that was actually read. Crawling only reads your public pages; your website is never changed. You can add more URLs (up to your memory limit) any time from the Knowledge screen for a deeper knowledge base.
File uploads
Upload documents directly. This works well for product manuals, policy documents, and internal guides that aren't published online.
- Documents:
.pdf,.docx,.doc,.txt,.md,.rtf,.epub - Spreadsheets:
.xlsx,.csv - Slides:
.pptx - Structured:
.json,.xml,.html - Images:
.jpg,.png(text is extracted via OCR)
Upload limits and how big batches work
- Per file: up to 50 MB. Files over that are rejected before upload with a clear message; split the file into smaller parts and add each one.
- Big batches: drop as many files as you like at once. Uploads are sent in parts automatically (up to 25 files per part), with live per-file progress.
- If a part fails: files that already uploaded are kept and start training; only the failed files are listed, each with the reason, and a single button retries them. A failed upload is a technical problem on our side, not a plan limit, so you never need to upgrade to fix one.
Text input
Type or paste text directly into the knowledge editor. Use this for custom content that doesn't exist anywhere else. Internal policies, specific talking points, brand-voice guidelines, or product details you want your agent to know.
Q&A pairs
Add specific question-and-answer pairs to guarantee your agent gives exact responses for common queries. When a visitor asks a question that closely matches one of your Q&A pairs, the agent uses your provided answer.
- Great for FAQs, return policies, pricing questions
- Ensures precise, consistent answers
- Takes priority over scraped website content for matching questions
Memory limits by plan
Each plan has a maximum amount of knowledge your agent can store, measured in characters. Every letter, space, and punctuation mark counts toward your agent's knowledge cap.
For context: a typical web page is about 2,000 characters, a short PDF around 5,000, and a long blog article around 15,000.
| Plan | Memory limit | Roughly equivalent to |
|---|---|---|
| Free | 150,000 characters | ~75 web pages |
| Starter | 25,000,000 characters | ~12,500 web pages |
| Advanced | 50,000,000 characters | ~25,000 web pages |
| Pro | 200,000,000 characters | ~100,000 web pages |
Adding your first knowledge during setup
When you set up your first AI agent, the knowledge step gives you three equal ways in, all on screen at once so you never have to go hunting for the one that suits you:
- Website. Paste your domain and we read your core pages. Still the fastest route if you have a public site.
- Paste text. Drop in your FAQs, policies, product details, opening hours, anything you would tell a customer. Give it a name so you can find it later, or leave the name blank and we will file it as "Pasted notes".
- Upload file. PDF, Word, text, spreadsheets, slides and more. You can pick several at once.
Any one of them is enough to start. Everything you add during setup lands in Knowledge exactly as if you had added it later, so you can read, edit or delete it any time.
Trying it with sample content
If you have no site and no documents to hand yet, or you would rather not share them until you have seen the product work, choose Try it with sample content. It loads a small sample knowledge base for Northwind Coffee Roasters, a business we made up for exactly this purpose: opening hours, delivery, returns, subscriptions, wholesale and a handful of common questions.
It is clearly labelled as a demo. The source appears in Knowledge > Text as "Demo knowledge, Northwind Coffee Roasters (sample business)" so it is never confused with your own content, and you can delete it in one click the moment you add yours. Ask your AI agent about the roastery's opening hours or delivery times and you will see how it answers from a real knowledge base before you commit any of your own.
Managing your knowledge
Adding sources
- Open Knowledge. In your agent sidebar, click Knowledge.
- Choose a source type. Select from Website, Files, Text, or Q&A at the top of the panel.
- Add your content. Enter a URL, upload a file, type text, or create a Q&A pair.
- Save. Your agent immediately learns from the new source. No separate training step required.
Editing sources
Click on any existing knowledge source to edit its content. Changes take effect immediately. For website URLs, you can re-crawl the page to pull the latest content.
Deleting sources
Click the delete icon next to any source to remove it. Your agent will no longer reference that content when answering questions. Deleting a source frees up memory toward your plan limit.
How the agent uses your knowledge
On every reply, your agent embeds the visitor's message, retrieves the top-matching chunks from your knowledge index, and uses those chunks as the source of truth for the answer. It does not reach into general internet knowledge to guess about your product. If the chunks don't cover the question, it says so honestly rather than making something up.
What it says when it doesn't know
When your knowledge doesn't cover a question, your AI agent says it doesn't have that information and offers to help with something else. It will not guess, and it will not fill the gap from general internet knowledge.
Just as importantly, it says that without describing how it works. Your visitors never hear about a knowledge base, training, sources, or anything else behind the scenes, in any language. Those words belong to your dashboard, not to a customer on your website, and hearing them makes a working AI agent read as a broken one. So you get this:
rather than a reply that mentions what the agent was trained on or where it looks things up. This is baked into the core prompt and can't be turned off. If you want a specific fallback wording or a handoff instead, set it in Agent behaviour, and if you would rather a human took over at that point, see Inbox.
If you are seeing this reply more often than you expect, it usually means retrieval is coming up empty rather than that anything is broken. Check that the relevant page or file is listed and trained under Knowledge, and add the missing content as a Q&A pair or a text snippet if a specific question keeps coming up short.
Auto-update knowledge
Advanced and Pro plans can re-crawl website sources on a schedule, so your agent stays current without manual re-training. Pick a cadence in the Training summary sidebar under Auto-update.
| Plan | Auto-update cadence |
|---|---|
| Free | Not available |
| Starter | Not available |
| Advanced | Weekly or Monthly |
| Pro | Daily, Every 3 days, or Weekly |
Troubleshooting
Training is stuck, or a crawl came back empty
Symptom. A website source sits on "Training" and never finishes, or it completes but adds little or no text and the agent says it doesn't know things that are clearly on the page.
Cause. The crawler reads the visible text a page returns. It comes back thin when the content is rendered entirely by JavaScript after load, sits behind a login or cookie wall, is mostly images or PDFs with no underlying text, or the seed URL points at a page whose real content lives on other paths. A crawl can also look "stuck" while it is still working through a large site.
What you'll see. As the crawl runs, the page count climbs live so you can see it working rather than watching a silent spinner. When a crawl finishes with almost no readable text, the source is marked Needs attention instead of appearing as a successful crawl. The Website page shows a warning ("We could not read content from this site") with a one-click Retry crawl button and shortcuts to add the content another way (another URL, a file upload, or pasted text). During your first crawl, if it comes back with barely a page, you'll see a recovery prompt right there with one-click options to add a specific page, upload a document, paste your text, or try the AI agent anyway, so you are never quietly handed an agent that has nothing to answer from. The Publish page also warns you before you go live if your AI agent's knowledge base is still nearly empty. If the crawl did capture a little text, the sandbox answers from that content rather than telling you it has no knowledge, so what you test is what your visitors would get.
Fix. Give it a minute on large sites, the status updates as pages stream in. If it finishes thin, use Retry crawl on the warning, point the seed URL straight at the page that holds the content (for example example.com/docs rather than the homepage), or add that content directly as a file upload, text, or Q&A pair so it does not depend on the crawler. When a site publishes its page list in a sitemap that we did not reach on the first pass, a retry can pick up those pages automatically. Remember the crawler skips low-value paths like /blog and /legal by default, so seed those explicitly if your product lives there.
The site blocked our crawler, so only the first page trained
Symptom. A website source is marked Needs attention with the message "This site blocked our AI agent's crawler, so we could only read its first page", even though that first page has plenty of content.
Cause. Some sites refuse automated readers (a bot-protection service, or a rule that blocks non-browser requests). When that happens, the crawler cannot follow links past the homepage, so it reads the seed page alone rather than the whole site. We now flag this instead of showing a successful crawl, because your AI agent would otherwise answer from a single page while looking fully trained.
Fix. Point the seed URL at a section that allows crawling (a docs, help, or landing subdomain often works), or add the rest of your content directly as file uploads, text, or Q&A pairs so it does not depend on the crawler. If you control the site's bot-protection settings, allowing verified crawlers lets a Retry crawl read the full site next time.
The site's security layer blocked the crawl completely
Symptom. A website source is marked Needs attention with the message "This site's security layer blocked our crawler. Try another site, or upload a document instead." There is no Retry crawl button on this warning.
Cause. Some sites sit behind a security service that shows every automated visitor a verification page (often titled "Just a moment...") instead of the real content. When that happens, even the first page the crawler reads is the verification screen, not your site, so there is genuinely nothing for your AI agent to learn from. We stop here on purpose: retrying the same address hits the same wall, and earlier versions of this flow could quietly train an agent on the verification page itself, which left it unable to answer anything about the business.
Fix. Add your content a way that does not depend on the crawler: upload a PDF or document, paste text, or add Q&A pairs, the agent learns from those just as well as from pages. A different address on the same business can also work when it is served outside the security layer (a docs, help, or landing subdomain often is). If you control the site's security settings, allowing verified crawlers lets a future crawl of the site succeed.
We could not connect to the site at all
Symptom. A website source is marked Needs attention with the message "We could not connect to this site, so we never reached your pages", followed by a list of addresses to allow. The site loads perfectly well in your own browser.
Cause. This one is not about your content, and it is not the same as being blocked. In the cases above the site answered us and refused; here it never answered at all. The connection timed out, which almost always means a firewall or security module on the site's hosting is dropping traffic from the addresses our crawler runs on, often before the site's own software ever sees the request. Server firewalls on managed hosting (Plesk, cPanel, or a security add-on such as Imunify360) commonly block whole ranges of data-centre addresses by default.
Fix. Ask whoever manages your hosting to allow these addresses, then use Retry crawl:
178.62.126.129178.62.118.239139.59.169.205161.35.172.746.101.81.65
Allow all five. Crawls run from any address in that pool, so allowing only one still leaves most crawls blocked. If your host cannot allow them, add your content as file uploads, pasted text, or Q&A pairs instead, which does not depend on reaching the site.
Every crawled page came back identical (the site redirects to one page)
Symptom. A website source is marked Needs attention with the message "Several pages on this site redirect to the homepage and drop the address" or "Most crawled pages returned identical content". An earlier crawl may instead show many pages that all carry the same title and the same character count, while your AI agent only ever answers from the homepage. Individual rows that bounced this way are marked Redirected and are not trained.
Cause. Some sites redirect every address to a single page, most commonly a redirect from the bare domain to the www address that drops the rest of the link. Every page the crawler asked for bounced to the same place and answered as though it were that page, so instead of your real pages it read one page many times.
What we do about it now. The crawler records where each page actually landed, not just the address it asked for. When a page's address is dropped and the site answers with its homepage instead, that page is skipped rather than saved as another copy of the homepage, and it costs none of your character allowance. Redirects that keep the address are ordinary and are still saved normally, so moving from http to https, from the bare domain to www, or between addresses with and without a trailing slash all continue to work.
Fix. Crawl the site's full address, including www if that is where your site actually lives (for example https://www.example.com rather than https://example.com), then use Retry crawl. If the flag persists, point the seed URL straight at the section that holds your content, or add it directly as a file upload, text, or Q&A pairs. If it still does not look right, contact support, this flag also alerts our team.
The crawl found your pages but read only a few words from each
Symptom. A website source is marked Needs attention with the message "We read only a few words from most pages on this site". The page count looks right and every row shows as trained, but the character count per page is tiny and your AI agent cannot answer anything specific about pages you know are full of text.
Cause. Some website builders hide sections of a page until you scroll down to them, as part of a fade-in or slide-in animation. Until that moment the browser treats the text as invisible, and a crawler reading the page the way a browser sees it would take only the part above the first screen. Dante now reads every page a second way, straight from the page's own markup, and keeps whichever reading captured more, so animated sections are no longer missed. Separately, a crawl whose pages are each nearly empty is now flagged rather than presented as a success, even when the pages add up to enough text overall.
Fix. Use Retry crawl. Sources crawled before this improvement keep the text they captured at the time, so an older website source needs one fresh crawl to pick up the full content. If the pages still come back nearly empty afterwards, the text is most likely inside an image or a video rather than written on the page: add it as pasted text, a file upload, or Q&A pairs, or contact support.
Some pages were skipped as "not found"
Symptom. After a crawl, the Website page shows a note like "3 pages skipped: not found", and those rows carry a Skipped badge with the detail "This page shows a not found message instead of real content".
Cause. Your site linked to addresses that no longer exist, but instead of returning a proper error, they show a "page not found" message styled like a normal page. Left alone, those error pages would train into your AI agent's knowledge and could surface in its answers. Dante detects them at training time and leaves them out; they use none of your storage. Real pages that merely talk about errors (say, an article about handling 404s) are not affected, the check only skips pages whose title or content is the error message itself.
Fix. Usually nothing, the skip is doing its job and your agent trains on the real pages. If a skipped address should exist, fix the page or the link on your site, then select the row and click Train (or use Retry crawl) and it will be read again.
Some pages were skipped as "blocked"
Symptom. After a crawl, rows on the Website page carry a Blocked badge and a blocked filter chip appears above the list. If most of the crawl came back that way, the source is marked Needs attention with the message "<your site> sent us blocked pages instead of your content".
Cause. The site answered our crawler, but with a refusal rather than your content: a security check ("Just a moment..." or a captcha), a Forbidden page, a "too many requests" notice, a password or login wall, or a parked placeholder such as "Example Domain". Left alone, those pages would train into your AI agent's knowledge, and it would answer your visitors out of an error page without either of you knowing. Dante now recognises them at training time and leaves them out, so they use none of your storage and none of your character allowance. A genuinely short page is never affected: the check looks for the refusal itself in the page's title or opening text, so a short contact page, an opening-hours page, or an article that merely discusses these errors all train normally.
Fix. It depends which kind you got, and the message on the row tells you. For a security check or a Forbidden page, ask whoever manages your site or its security layer to allow our crawler, then use Retry crawl. For a "too many requests" notice, a retry on its own usually works. For a password or login wall, the page is not readable without signing in, so upload it as a document instead. For a placeholder, check the address is the one your visitors actually use.
Every page trained, but each holds only a line or two
Symptom. A website source is marked Needs attention with the message "Every page we read from this site stored only a line or two of text", even though several pages were saved and the total looks reasonable.
Cause. A handful of pages each holding a sentence adds up to a total that looks healthy while no single page can actually answer a question. This usually means the pages load their real content with JavaScript after the initial response, or the addresses that were crawled are navigation stubs rather than the pages that hold your writing. One page with real content is enough to clear this warning, so it only appears when nothing in the whole crawl carries a proper answer.
Fix. Point the seed URL straight at a page that holds real written content (for example example.com/docs rather than a landing page), or add that content directly as a file upload, pasted text, or a Q&A pair so it does not depend on the crawler reading your pages.
Some pages were blocked by your site
Symptom. After a crawl, the Website page shows a note like "12 pages blocked by your site", and those rows carry an amber Blocked badge with their own filter. The titles on those rows often read "Just a moment...", "403 Forbidden" or "429 Too Many Requests" rather than your real page titles.
Cause. This is not a problem with your pages. Your hosting, or a security service in front of it, treated our crawler as automated traffic and served it a security or error page instead of your content. Because that page arrives as a normal successful response, a crawl like this used to look like it had worked: the rows said trained, and your AI agent then answered visitors from the security page instead of from your site. Dante now recognises those pages at training time and leaves them out, so they use none of your storage and never reach your agent's answers. Real pages that merely discuss security or errors are not affected, the check only skips pages that are themselves the block page.
Fix. Ask whoever manages your hosting to allow our crawler's addresses, then select the blocked rows and click Train (or use Retry crawl). The addresses are listed in the note on the page, and all of them need allowing rather than just one, because crawls run from any of them. If you would rather not change your hosting, add your key pages as a file upload, pasted text, or Q&A pairs instead.
A file upload failed, or only some files went through
Symptom. After dropping a batch of files, some rows show a red error while others uploaded fine.
Cause. Either a file is over the 50 MB per-file limit, or the upload service had a temporary problem sending one of the parts. Uploads are sent in parts automatically, so one failed part never takes down the rest of the batch.
Fix. Files that uploaded are already saved and training; you do not need to re-add them. For a file over 50 MB, split it into smaller files and add those. For anything else, use the Retry failed files button on the upload panel. This kind of failure is technical, not a plan limit, so upgrading your plan will not change it; if retries keep failing, contact support.
"We couldn't read your site" during onboarding
Symptom. While setting up your first AI agent, the add-your-site step reports that your site couldn't be read, and offers to try a different URL, upload a document, or continue anyway.
Cause. Some sites protect themselves from automated readers (for example with a bot-protection or challenge page). The crawler receives that protection page instead of your real content, so there is nothing for your AI agent to learn from.
Fix. Try a URL that is not behind the protection (a docs, help, or landing subdomain often works), or switch to the Upload file or Paste text tab right there on the same step, the agent learns from a document or from pasted text just as well as from pages. You can also continue and add knowledge later from the dashboard under Knowledge. If your site uses a bot-protection service you control, allowing verified crawlers in its settings lets future re-crawls succeed.
The setup step now tells you exactly what went wrong, rather than leaving you watching a progress bar:
- "We could not connect to [your site]". We never reached your pages at all, so this is not a problem with your content. A firewall or security rule on your hosting is dropping our crawler. The message lists the addresses to allow, and all of them have to be allowed because crawls run from any one of them. Once they are, use Try again.
- "The security layer on [your site] blocked our crawler". Your site served a security challenge instead of your content. Retrying the same address hits the same wall, so no retry is offered and neither is continue-anyway: an agent trained at that point would have nothing at all to answer from. Pick another site, upload a document, or paste your text.
- "[your site] refused our crawler". The site turned us away, so we could only see its first page. A retry can work once the site allows us.
- "We could not read any pages from [your site]". The crawl finished without storing anything. Add a specific page, upload a document, or paste your text.
- "Your knowledge base looks thin". We read your pages but stored very little text, often because the site builds its content with JavaScript that our crawler cannot see. Your AI agent does have a little to work with, so you can add more or try it anyway.
You are never stuck on this step. Every state except the fully blocked one offers Try my AI agent anyway, and Skip onboarding stays available throughout.
Still stuck? We'll help.
Every plan, including Free, comes with human support. If something isn't behaving the way you expect after following the steps above, send us a note and we'll get back to you promptly.