Grounded AI vs RAG: what actually stops hallucinations

RAG is how an answer is retrieved. Grounding is a rule about what the AI is allowed to say. Retrieval augmented generation searches a source, puts what it finds into the prompt, and generates an answer. Grounding adds three constraints on top: answer only from that material, show where it came from, and refuse when it does not cover the question. Every grounded system uses retrieval. Not every system using retrieval is grounded.

That distinction sounds academic until a customer is told the wrong refund window.

What RAG actually does

Without retrieval, a language model answers from whatever it absorbed during training. It has never seen your shipping policy, so when asked about it, it produces something shaped like a shipping policy. Fluent, confident, invented.

RAG fixes the input problem. Your content is split into passages and indexed. When a question arrives, the system finds the passages that look most relevant and puts them in the prompt alongside the question. The model now has your actual policy in front of it.

This is a genuine improvement and most AI support tools do it. It is also where most of them stop.

Why that is not enough

Nothing in RAG forces the model to stay inside the material it was given. The retrieved passages are context, not a fence. Three things go wrong in practice.

Retrieval comes back weak. The customer asks something your site does not cover, so the search returns the closest passages rather than the correct ones. The model, handed vaguely related text and a question, answers anyway. It has no way to signal that the material was a poor match.

The model fills gaps. Given a passage covering four of five parts of a question, a model will often complete the fifth from general knowledge, in the same tone, with no seam. You cannot see where your content stopped and the invention began.

The question is misunderstood. Follow-ups are where this bites. "Does it come in blue?" retrieves on that sentence alone, and the pronoun has no referent, so the search returns something confidently unrelated. The conversation history in the prompt then makes the reply read as though the assistant understood perfectly.

The three things grounding adds

Grounding is not a better retriever. It is a set of rules enforced around the model.

A closed source. The answer may only be constructed from retrieved material. Not the open internet, not the model's training data, not a blend. If it is not in your content, it is not available to say.

An attributable answer. Every reply carries the page it came from, shown to the person reading it. This does more than reassure. It makes the system auditable: a wrong answer stops being a mystery about the model and becomes a specific page that needs fixing.

A refusal path. When retrieval finds nothing relevant enough, the system does not answer. It says so and offers a person. This is the part vendors skip, because a tool that sometimes says "I do not know" demos worse than one that always has something to say.

That third rule is the one doing the real work. Grounding is defined by what it will not do.

How to tell which one you are looking at

Vendor pages use the words interchangeably, so test rather than read.

• Ask something your content does not cover. A grounded system says so and offers a human. A RAG-only system answers anyway. This single test separates them.
• Check for a source under every reply, not just some. Occasional citation means citation is decorative.
• Ask a follow-up with a pronoun. "Does it come in blue" after a product question. If the answer changes subject, retrieval is running on the raw sentence with no context.
• Ask what it has read. If you cannot see and correct the indexed content, you cannot fix a wrong answer at its cause.
• Ask what happens at low confidence. If there is no answer to that question, there is no floor.

The cost nobody mentions

Grounding will answer fewer questions than an ungrounded system. That is not a defect, it is the trade, and it is worth being clear-eyed about it. You are choosing a smaller number of answers you can stand behind over a larger number you cannot.

For a demo, the ungrounded system wins. For a business where a wrong delivery date is a refund and a wrong medical statement is far worse, the trade is not close.

And the gap closes on its own, because every refusal is logged as a question your content did not answer. Write those pages and coverage rises, on a foundation that stays true.