You opened a transcript, or a customer email, and there it was: your website chatbot giving customers wrong answers. Maybe it quoted a price you do not charge, invented a refund rule, or promised a service you have never offered. The bot is still live, still answering, and it may be doing the same thing right now while nobody is watching. The worry underneath is real: a customer who acted on a bad answer, a refund you did not plan for, and a question about whether you have to honor what the bot said. You can stop new damage today without a rebuild and without a vendor call. The first moves take less than an hour, and switching part of the bot off is one of them. It is a normal safety decision, and you are allowed to make it.
Key Takeaways
Take prices, refunds, policies, availability and bookings away from the chatbot, or switch its generated answers off, until each answer comes from a source you have written and checked.
Save each bad conversation, note what the customer was told and whether they acted on it, and send them the correct answer yourself.
NIST calls confidently stated false content "confabulation," and research on retrieval-augmented answers still found unsupported and contradictory claims.
In Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for its chatbot's wrong answer and ordered it to pay CAD $650.88 plus interest and fees.
Pull the Bot Back From Prices, Policies and Bookings Today
The fastest fix is to narrow what the chatbot is allowed to say. Anything with money, eligibility or a promise attached comes out of the bot's hands first, because those are the answers a customer can act on. The steps below work in roughly this order, and none of them needs a developer.
Save the evidence before you change anything
Export or screenshot the bad conversation. Write down the exact question, the answer the bot gave, the correct answer your business stands behind, and whether the customer bought, booked or cancelled because of it. Do this before you edit settings, so you still have the original record. You will need this record to fix the source and to reply to the customer.
Turn off the topics that carry risk
Go into your chatbot's settings and remove or restrict answers about prices, fees, refunds, cancellations, eligibility, service area, availability, bookings and anything contractual. If your platform lets you, route those topics to a person instead. On Microsoft's platform, for example, makers can turn off a setting called "Allow ungrounded responses," which keeps answers inside the sources you configured, according to its FAQ for generative answers. If you cannot find a clean way to restrict those topics, switch the generated answers off for now.
Tell affected customers the correct answer
Reach out to each customer the bot misled, in your own words, with the right price, policy or date. Do not wait for them to find out at checkout or on the phone. How you handle what the bot promised is a business decision, and if a specific customer is disputing it, that part needs advice for your own situation and location rather than a chatbot setting.
Switching It Off Is the Right Call in Some Cases
We build chatbots, so it would be easy to talk you out of this. We will not. Turning the chatbot off, at least for a while, is the right move when any of the following is true:
- It makes unsupported promises: It is stating prices, eligibility, refunds, availability, contract terms or regulated advice that you cannot correct quickly.
- There is no current approved source: Nobody has written and checked the answers it should be drawing from.
- The human handoff does not work: A customer who asks for a person gets nowhere, or reaches an inbox no one reads.
- Nobody reviews the transcripts: The bot is answering and no one on your team is reading what it says.
Replace it the same day with something that cannot invent an answer. That can be a plainly labeled contact route, your published hours, a short request form that collects the question and promises a reply from a person, and a small FAQ that contains only facts you have approved. For lead capture, a form that says "we will get back to you" is better than a bot that makes something up to look helpful.
Bring it back only when four things are in place: someone owns each policy source, the bot refuses questions it should not answer, a person can take over, and it has passed a written test. Pausing a feature that is misleading customers is a control, the same as pulling a wrong price off a shelf. It is not a verdict on your business.
Your Chatbot Guesses Because That Is How It Works
A generative chatbot does not look up your refund rule the way a database would. It predicts a plausible next response from patterns, and a plausible answer can be wrong. The NIST generative AI profile, published in July 2024, names this "confabulation," a system generating and confidently presenting false content, and describes it as a natural result of how these models approximate patterns in their training data.
That explains the tone that makes these errors hard to catch. The bot sounds just as sure when it is wrong as when it is right, so a confident answer tells you nothing about whether it came from a checked fact.
Why a knowledge base did not stop it
Connecting the bot to your documents, often called a knowledge base, gives it better material to draw from. It reduces the problem. It does not remove it. The 2024 RAGTruth research corpus documented unsupported or contradictory claims even in answers that were built from retrieved sources. If you want the background on what a knowledge base does well, our post on what an AI knowledge base is covers it.
What in your setup makes it worse
Some causes sit in the configuration, and you can check them today:
- Ungrounded answers are allowed: Microsoft documents a setting that permits the bot to answer beyond your configured sources.
- The same question gets different answers: Intercom says answers can vary because different sources or models may be used at different times, in its Fin AI Agent FAQs.
- Website content updates on a delay: Intercom notes that external website content can refresh weekly, not instantly, so a price you changed on Monday may still be the old price in the bot.
- Old and new pages disagree: Stale, conflicting or incomplete business content becomes a customer-facing answer.
Notice what those causes have in common. Each one is a content or settings problem you can find and correct, which is why the fix starts with your sources rather than with the bot's wording.
Each Kind of Wrong Answer Has Its Own Fix
Once the risky topics are paused, trace every bad answer back to a cause. No credible cross-vendor study ranks these failures by how often they happen, so the order below follows consequence, not frequency. Price, refund, availability and eligibility answers go first because a customer can act on them.
| What went wrong | Type of job | What the fix involves |
|---|---|---|
| Old or conflicting policy content | Small content job | Name an owner, correct the approved source, remove duplicates, date the update, retest |
| Answers outside what it should cover | Configuration job | Limit sources, turn off ungrounded answers, add refusal wording, route sensitive topics to staff |
| It misread a customer's wording | Moderate content and testing job | Add exact FAQs, synonyms and short policy snippets, test with real customer phrasing |
| Generated answers clash with scripted flows | Implementation job | Review how topics are routed so only one answer path fires |
| The handoff to a person fails | Support operations job | Check triggers, destination, hours, transcript transfer and the failure message |
| It quotes, books or refunds on its own | Larger integration job | Answers must come from a confirmed result in your booking or billing system, not model text |
The vendors document the controls that sit behind these fixes. Intercom says its agent can answer only from content you have added, ask a clarifying question when a request is unclear, say it found no answer, and hand the conversation to a person. Its answer debugging tool shows which source the bot used, so you can correct that source or add a short snippet. Microsoft's fallback topic can pass a conversation to a person after the bot fails to understand a request.
None of these is a complete fix alone. The protection comes in layers: scoped content, firm refusal rules for prices, refunds, legal or medical questions, a working handoff, saved transcripts, regular review, and a retest every time a price or policy changes. A better prompt by itself does not cover that list.
If the bot needs to quote live prices or take bookings, the answer has to come from your actual system, which is an AI integrations job rather than a wording change.
A Fifteen-Minute Test Shows You Where It Breaks
You can test your own chatbot today, and you should. Open a fresh visitor session, such as a private browser window, so the bot has no memory of you. Keep a one-page sheet of the correct answers beside you, and record every question and reply.
Ask these ten questions, in your customers' own words:
- What is the current price for your most common service or product?
- Does tax, shipping, setup or any fee apply?
- What are your business hours?
- Are you open on holidays or after hours?
- What is your refund or cancellation rule?
- Who is eligible, or which areas do you serve?
- Does a specific current promotion apply?
- How do I book or change an appointment?
- Is a specific item or time slot available?
- How do I reach a real person?
Then ask one question the bot should refuse, such as "Can you guarantee a full refund after 90 days?" The safe result is a refusal or a handoff to a person. A confident, plausible answer is a failure, even if it sounds polite.
Mark every reply that is unsupported, outdated, contradictory or unhelpful, and pause those topics until the source is fixed. Run the same test weekly, and again after any change to prices, hours or policies, because the bot can drift when its sources change. If you want a record you can compare over time, keep the answer sheet and the results together in one folder.
A Business Can Be Held to What Its Chatbot Says
This part is general information, not legal advice. The clearest documented decision is Moffatt v. Air Canada, 2024 BCCRT 149, decided on February 14, 2024 by British Columbia's Civil Resolution Tribunal. Air Canada's website chatbot told a customer they could apply for a bereavement fare refund after travel, although the airline's linked policy said otherwise.
The tribunal found negligent misrepresentation. It held Air Canada responsible for information on its website, whether that information appeared on a static page or came from the chatbot, and awarded the fare difference of CAD $650.88 plus interest and tribunal fees. Air Canada tried to distance itself from the chatbot, and that did not change the outcome.
Keep the limits in view. This is a small-claims tribunal decision in one Canadian province, not a universal rule, and it does not tell you what would happen in your location. What it does show is that you should not assume your chatbot is legally separate from your business. Treat everything it says as something your business said.
Two other points affect what you do next. A disclaimer on the chat window may inform customers, but it does not make a false statement from your business disappear. And the U.S. Federal Trade Commission said in its February 27, 2023 guidance, keep your AI claims in check, that there is no AI exemption from existing laws. If you serve customers in the EU, the European Commission describes the AI Act as requiring that people be clearly told when they are interacting with a chatbot, subject to the law's scope and timing. None of these sources hands you a checklist that proves a particular chatbot is compliant. Together they support the same practical steps: disclose the bot, keep it to claims you can back up, and keep watching what it says.
A Well-Run Chatbot Looks Different Day to Day
You can feel the difference between a chatbot that is under control and one that is not. A well-run one has one named owner for each policy source, a clear list of what it answers, current content, a refusal route for unknown or sensitive questions, a handoff that works, and someone who reads the transcripts on a schedule.
What changes for you is concrete. Staff stop correcting the bot's answers. Customers stop calling back because the self-service answer was wrong. When a policy changes, you catch the broken answer in the next test instead of in a complaint. A risky deployment, by contrast, is judged mainly by how many conversations it answered, not whether the answers were right.
No vendor publishes a cross-platform accuracy rate that proves one setup is safe, and no one can promise a chatbot that never makes things up. So, judge yours by the test results and the transcripts, not by a feature list.
Is your chatbot safe to leave on?
Pick an answer to begin.
1. Your chatbot is connected to a knowledge base. Does that stop it from giving wrong answers?
2. During the fifteen-minute test, you ask whether a full refund is guaranteed after 90 days. What is the safe result?
3. The bot has no current approved source and no working human handoff. What should you do?
Frequently Asked Questions About chatbot giving customers wrong answers
Why is my AI chatbot giving wrong information?
A generative chatbot predicts a plausible response instead of checking your business rules, so it can state false things with confidence. NIST calls this confabulation. Stale, conflicting or missing content in its sources, and settings that allow ungrounded answers, make it more likely.
Should I turn off a chatbot that gives wrong answers?
Pause the affected topics, or the whole generated-answer feature, when it makes unsupported claims about prices, refunds, eligibility or bookings, or when there is no working handoff to a person. That is a normal safety decision.
Who is liable for chatbot answers?
This is general information, not legal advice. In Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for its chatbot's wrong answer. It is one small-claims decision, not a universal rule, so get advice for your own location if a customer is disputing something.
How do I fix chatbot answers that keep coming out wrong?
Trace each bad answer to its source, correct or remove the conflicting content, limit the bot to approved sources, turn off ungrounded answers where your platform allows it, and add refusal and handoff rules for sensitive topics. Then retest.
How do I test an AI chatbot?
Open a fresh session, ask ten real customer questions about price, fees, hours, refunds, eligibility, promotions, bookings, availability and reaching a person, then ask one question it should refuse. Compare every reply with your approved answer sheet.
Does chatbot hallucination in customer service mean the tool is useless?
No. It means the bot needs bounded questions, current sources, refusal rules, a working handoff and regular review. Some tasks, such as quoting live prices or issuing refunds, may be better handled by a person or a fixed flow.
Wrapping Up
When your chatbot is giving customers wrong answers, the first job is containment: take prices, policies, availability and bookings out of its hands, save the bad conversations, and tell affected customers the right answer. The errors happen because the bot predicts plausible replies rather than checking your rules, and a knowledge base narrows that gap without closing it. The Moffatt decision shows a business can be held to what its bot said, so treat every answer as your own. And if the bot has no checked source, no handoff and no one reading its transcripts, switching it off is the correct call.
A chatbot that answers only from checked sources, hands off what it should not touch, and passes a weekly fifteen-minute test can take routine questions off your team's plate without putting a wrong price in front of a customer.
At Web Leveling, we build and repair website chatbots with scoped sources, refusal rules, working handoffs and a written test, and we will tell you when a contact form or a short FAQ is the better choice for your business. We work with small and medium businesses across the country and overseas. Send us the transcript that worried you, and we will help you work out what caused it and what to do next.
Terms
Chatbot words in this post
Tap a term to see what it means.
Generative chatbot. A chat assistant that writes its own replies by predicting likely text, rather than choosing from fixed answers.
Confabulation. NIST's term for an AI system generating and confidently presenting false content, often called hallucination.
Knowledge base. The set of documents and pages a chatbot is allowed to draw its answers from.
Ungrounded response. An answer the bot gives without support from the sources you configured.
Human handoff. The step where a chatbot passes the conversation to a person on your team.
Refusal rule. An instruction that makes the bot decline a question and route it elsewhere instead of guessing.