Chatbots that answer only from your documents: how grounding works, and how to test it

I spent two decades hiring people, and the most dangerous candidate was never the one who said "I don't know." It was the one who answered every question with total confidence and was quietly wrong about half of them. You only find out later, in the room where it costs something. A chatbot that makes things up is that candidate, standing on your website, answering your customers in your name, around the clock.

That is not a rare failure. A 2024 Stanford study tested the leading AI legal research tools, the ones sold to lawyers with promises of reliable answers, and found they still "hallucinate between 17% and 33% of the time." These were not toy chatbots. They were built to pull from real legal sources, and they still invented things a third of the time. So the honest question is not whether an AI can sound sure of itself. It is whether it can show you where its answer came from, and admit when it cannot.

What is a chatbot that answers only from your documents?

A chatbot that answers only from your documents is a grounded chatbot: before it replies, it searches your own approved content, pulls the specific passage that answers the question, and shows you the source. When your documents do not cover the question, it says so instead of guessing. The plain version: it speaks from your pages, not from its memory, and it can prove it.

The method behind this has a name. It is called retrieval-augmented generation, or RAG, and it was introduced in a 2020 paper by Patrick Lewis and colleagues. Their finding was simple and has held up: models that retrieve from a real source "generate more specific, diverse and factual language" than models answering from memory alone. Three years later, a large survey of the field by Gao and colleagues put the business case in one line: pulling answers from an external source "enhances the accuracy and credibility of the generation" and lets the knowledge stay current as your content changes. Grounding is not a gimmick. It is the accepted way to keep an AI honest.

Why generic chatbots invent answers

To see why grounding matters, look at what a generic chatbot is actually doing. It is not looking anything up. It is predicting the next few words based on patterns it absorbed in training, and when the pattern runs out, it fills the gap with something that sounds right. Researchers have a precise word for this. A 2023 survey of hallucination by Huang and colleagues describes it as generated content that "appears nonsensical or unfaithful to the provided source content." In plain terms: the bot says something that either contradicts your material or cannot be checked against it at all.

A generic chatbot

Answers from memory. Asked something your documents do not cover, it guesses, and the guess arrives in the same confident tone as a fact. There is no source to click, because there was no source. You find the error later, in front of a customer.

A grounded chatbot

Answers from your content. It searches first, pulls the passage, and puts the citation right there. Asked something your documents do not cover, it declines and hands the visitor to a person. You can trace every answer back to the page it came from.

That difference is the whole game. A wrong answer with no source is a liability wearing a friendly face. A cited answer, or an honest "I don't know," is customer service you can stand behind.

Why a refusal is a feature, not a failure

Here is the part people get backwards. They judge a chatbot by how much it can answer, when the number that actually protects you is how gracefully it refuses. In my judgment, the refusal is the single most underrated feature in this whole category.

Think about the hiring bar again. The standard you want in the room is not the person who always has an answer. It is the person you can trust when they do have one, because they will tell you plainly when they don't. A chatbot earns that same trust the same way. When it declines a question it cannot ground, it is not failing. It is telling you the truth about the edge of its knowledge, which is exactly what you want it doing before it says something wrong in your name.

A confident wrong answer is not a small bug. In a regulated business, it is an incident report waiting to be written.

The standard worth holding any grounded chatbot to is short: cite or decline. Every answer gets checked against the approved pages before it goes out, and when nothing supports a question, the bot says so instead of inventing an answer. A confident wrong answer in your name costs more than a missing one. That is not a slogan. For a bank, an advisor, a clinic, or a school, one invented answer is a compliance event, and everyone on the team knows it.

Five ways to test any chatbot's grounding claim

Every vendor in this space will tell you their bot is grounded and cited. One grounded support vendor, Wonderchat, states it flatly on its own site: "Every response cites its source." Some vendors are telling the truth. The only way to know which is to test the claim yourself, and it takes about five minutes. Here is the standard I would hold any chatbot to before it goes anywhere near a customer.

1. Ask it something your documents cannot answer. This is the most important test, so run it first. A grounded chatbot should decline and route you to a person. A weak one will invent a confident answer out of thin air. If it makes something up here, nothing else on this list matters.

2. Check that the citation actually contains the answer. A shown source is not the same as a real source. Click it. Read it. Does the passage it points to actually support the answer the bot gave? Remember the Stanford number: even tools built on retrieval were wrong up to a third of the time, so the citation being present is not proof the citation is right.

3. Ask the same question two different ways. Grounding should hold up when you rephrase. If a plain question gets a clean cited answer but a slightly reworded one gets a vague guess, the grounding is thinner than advertised.

4. Ask who is responsible when it is wrong. This is a people question, not a software one. Find out who reviews what the bot says, who gets alerted on a refusal, and who owns the fix. A bot with no human accountable behind it is a rehearsal with no director. The answer should be a named process, not a shrug.

5. Ask to see the record. A serious grounded chatbot logs every question and every citation it returns, so that when compliance or legal asks what it said and why, you can show them, exactly. If a vendor cannot produce that trail, the grounding is a claim, not a system.

Run those five, and the marketing melts away fast. What is left is whether the thing actually answers from your documents, and admits when it can't.

Plug-and-play or governed: which grounded chatbot fits?

Here is the distinction most vendor pages skip, and it decides more than any feature list. Grounded chatbots come in two shapes, and they are not the same product at two prices. They are two different answers to one question: who owns the wrong answer?

Plug-and-play grounded software is something your team sets up and runs. You point it at your content, drop the widget on the site, and manage it yourself. It is built for organizations that have the staff to own their own support tooling, that want speed and broad channel coverage, and whose wrong answers are recoverable. For that buyer, a good grounded product does a real job well. But the work stays yours: you tune the grounding, decide what happens at the edge of its knowledge, and answer for it when it slips.

Grounding is not a promise the bot is always right. It is a promise the bot can always show its work, and knows when to stay quiet.

A governed deployment inverts that. A team stands the agent up on your site, tunes the grounding to your industry, and stays accountable for it. The difference is not the grounding itself; it is the layer wrapped around it for places where a wrong answer is not a bad review, it is a filing. That layer is where you should look hard, because it is what separates a demo from a system a regulated business can stand behind:

Cite or decline as a designed step, not a hope. The bot returns a sourced answer or an honest refusal, never a confident guess.
Per-industry rule packs that attach the required disclaimers before a person ever sees the answer, so a bank's bot and a clinic's bot are not held to the same generic script.
Crisis routing to a human. Anything that reads like an emergency goes to real help immediately, not into a chat queue.
A full audit trail. Every question and every citation is logged, so when compliance or legal asks what the bot said and why, you can show them, exactly.
A named human in control of everything the agent publishes, not just in the loop watching it.

Neither shape is the right answer for everyone. If you have the people to run your own tooling and your wrong answers are recoverable, plug-and-play fits. If you are a bank, an advisor, a clinic, a school, or an agency's regulated client that cannot afford a made-up answer, the grounding is table stakes and the governance is the point. Know which buyer you are before you compare anything else, because a bot built for the other one will fail you no matter how well it cites.

2020

the year researchers named RAG

Lewis et al., the original paper

17-33%

how often leading legal AI tools still made things up, even with retrieval

Magesh et al., Stanford, 2024

Where this is heading

Grounding is going to become table stakes. A year ago, "answers only from your documents" was a differentiator you could charge for. Soon every serious chatbot will claim it, the way every car now claims to have brakes. When that happens, the claim stops meaning much, and the real question moves to what the bot does at the edge of its knowledge: does it decline, does it log the exchange, and is there a named human standing behind it who answers for the fix.

Based in Austin, Texas, our team builds for that edge on purpose, because it is where trust is either earned or lost. The goal was never a bot that always has an answer. It was a bot you can trust with the ones it gives you, and one honest enough to hand the hard question to a person. That is the standard. It is the same one I used in every hiring room for twenty years, and it turns out it is the right one for the machine on your website too.

About the practice behind this guide

This guide comes out of daily practice, not theory. Trinzik is a boutique studio in Austin, Texas: we build native, custom-code websites, run SEO and AI visibility programs, stand up compliance-grade grounded chatbots, and provide high-end editorial and creative support alongside digital marketing services. The five-test standard above is the one we hold our own grounded agents to, and our published case studies show the method applied.

Questions this raises

What does it mean for a chatbot to answer only from your documents?

It means the chatbot is grounded: before it replies, it searches your approved content, pulls the passage that answers the question, and shows the source. If nothing in your documents supports an answer, it says so instead of guessing. This is the difference between a bot that speaks from your published pages and a generic bot that answers from whatever it absorbed in training, with no way to show where a claim came from.

Does grounding, or RAG, stop AI from making things up completely?

No. Grounding lowers the risk a lot, because the model answers from a retrieved source instead of memory, but it does not reach zero. A 2024 Stanford study found leading legal AI tools built on retrieval still produced wrong or unsupported answers 17% to 33% of the time. That is why a refusal step matters, and why any vendor claim of zero made-up answers deserves a test, not your trust.

How do I test a chatbot's grounding claim before I trust it?

Ask it a question your documents genuinely cannot answer, and watch what it does. A grounded chatbot should decline and point you to a person. A weak one will invent a confident answer with no source. Then ask a question your documents can answer, and check that the citation it shows actually contains the answer. Two questions, five minutes, and you know whether the grounding is real.

Sources

← All dispatches