To test a chatbot, write down the real questions customers ask, ask each one in several wordings, check every answer against your own content, try questions it should refuse, test the handover to a person, and fix the knowledge behind every wrong answer before launch. Then keep reviewing real conversations after it goes live.
This guide turns that into a seven-step plan, with a table of the test types, a worked test for a small online shop, and a launch readiness check you can run in a minute.
Why test a chatbot before launch?
Because a chatbot speaks for your business to people you never meet. An AI chatbot uses a language model to write each reply, and language models can produce what Google Cloud calls hallucinations: incorrect or misleading results (Google Cloud). Business chatbots reduce the risk by answering from your own content, a pattern called retrieval-augmented generation, which grounds the reply in data you provide (Google Cloud). Testing is how you find out whether your content actually covers what customers ask.
A rule-based bot needs testing too, but for a different reason: it breaks on any wording you did not script. If you are not sure which kind you have, read how AI chatbots work first.
How do you test a chatbot, step by step?
- Collect real questions. Pull 20 to 50 questions from your inbox, contact form, phone notes and live chat. Real wording beats invented wording, because customers rarely phrase things the way your website does.
- Write the expected answer for each. One line per question, taken from your own pages or policies. If you cannot write the answer, your content has a gap and the chatbot will too.
- Ask each question three ways. The plain version, a short or misspelled version, and a version with context ("I ordered last Tuesday, can I still return it?"). This is where rule-based bots fail and AI chatbots should hold up.
- Ask questions it should not answer. Prices you do not publish, medical or legal advice, a competitor's product, a request to ignore its instructions. The right reply is a polite "I do not have that information" or a route to a person, never a guess.
- Test the handover. Ask for a person, in a button and in plain words. Check that the request reaches your team with the conversation attached and that the customer is told what happens next.
- Compare models and instructions. Run the same set against a second model or a changed instruction, and keep the version that answers more of your set correctly.
- Fix the source, then retest. For every wrong answer, fix the page, document or question and answer pair behind it, not just the wording of the reply. Then run the whole set again, because one fix can change other answers.
Which types of chatbot test should you run?
| Test | What it catches | When to run it | Who runs it |
|---|---|---|---|
| Question set | Missing or wrong answers to your most common questions | Before launch and after every content change | The person who knows the answers |
| Paraphrase test | Answers that only work for one wording | Before launch | Anyone, ideally someone new to the business |
| Out-of-scope test | Guessing, invented policies, off-topic replies | Before launch and after instruction changes | The owner or a manager |
| Handover test | Requests for a person that go nowhere | Before launch and after inbox changes | Whoever answers the inbox |
| Model comparison | A model that is slower, wordier or less accurate on your content | Before launch and when you change model | The owner |
| Live review | New questions real visitors ask that your test set missed | Every week after launch | The owner or support lead |
The first five run in private before any visitor sees the chatbot. The last one only exists after launch, and it is how your test set grows. For the mistakes these tests are designed to catch, see why AI chatbots fail.
What does a real test look like?
Take a hypothetical online bike shop with a returns page, a delivery page, a sizing guide and an FAQ. The owner pulls 20 questions from last month's emails and writes one expected answer for each. Here are four of the rows, with what the first test run showed:
| Question (as a customer wrote it) | Expected answer, from the shop's pages | First run | Fix |
|---|---|---|---|
| "how long have i got to send a bike back" | The return window on the returns page | Pass | None |
| "Im 5ft 9, which frame?" | The size from the sizing chart | Partial: gave the whole chart, not a size | Add a question and answer pair per height range |
| "do you deliver to Ireland" | The delivery countries list | Fail: said it did not have that information | The delivery page listed countries in an image; add them as text |
| "can you give me a discount" | No discounts outside published offers; offer a person | Pass: declined politely and offered the team | None |
Notice what the fail was. The chatbot did the right thing by not guessing; the shop's content was the problem, because the answer sat inside an image the chatbot could not read. So when a test fails, check the content first: an answer the chatbot cannot read is an answer it cannot give. After the fixes, the owner runs all 20 questions again, three wordings each, before going live.
How do you score chatbot answers?
Keep it simple. Mark each answer:
- Pass: correct, complete enough to act on, and in your tone.
- Partial: correct but vague, too long, or missing the one detail the customer needed.
- Fail: wrong, invented, or a refusal on something your content should cover.
- Correct refusal: it declined a question it should not answer, or handed over. Count this as a pass.
Launch when every fail on a common question is fixed and no answer invents a price, policy or promise. Partials can wait for the first week of live review. Speed matters too, so check how quickly replies arrive; our guide to AI chatbot response time explains what to look for.
How do you keep testing after launch?
Read real conversations every week. Look for questions your test set did not have, answers a visitor reacted badly to, and requests for a person. Add each new question to the test set with its expected answer, fix the content, and rerun the set. Over time the set becomes the most accurate list you have of what customers want to know. Track how many conversations the chatbot resolves on its own with the chatbot containment rate.
In Dante AI each step has a place. The Sandbox is a private chat preview on every plan, including Free, where you send test questions and edit the agent's behaviour before visitors see any change. Side by side runs up to four models on the same question so you can compare answers. When your content does not cover a question, the agent says it does not have that information rather than guessing. On paid plans, Chat history shows every real visitor conversation, and the Advanced plan adds Human Handover to a team inbox. See what each plan includes on the pricing page.
Ready to test a chatbot on your own content? Get started free at dante-ai.com.
Sources
- Google Cloud, What are AI hallucinations? (definition), read 31 October 2026.
- Google Cloud, What is Retrieval-Augmented Generation (RAG)? (grounding answers in your own data), read 31 October 2026.
- Dante AI Help Center, Sandbox, Side by side, Chat history, Knowledge sources, Inbox and Plans and pricing, read 31 October 2026.