RAG, Explained Honestly: What Retrieval-Augmented Generation Can and Can't Fix

"We use RAG, so it doesn't hallucinate" is a sentence you'll hear from a lot of AI product pages. It's worth separating what's true in that sentence from what's marketing, because the honest version is more useful — and still a real reason to prefer a system built this way.

What RAG actually is

A language model's "knowledge" is whatever got baked into it during training — frozen at a point in time, and with no way to point at a source for any given claim. Retrieval-Augmented Generation changes the shape of the problem: instead of asking the model to answer purely from memory, the system first searches a document collection — your own knowledge base, in an enterprise setting — for passages relevant to the question, and hands those passages to the model as part of its input. The model's job shifts from "recall this fact" to "summarize what these specific documents say," which is a task language models are demonstrably better at, and one that comes with an actual source to check.

That's a genuinely useful shift. It's also not the end of the story.

What the research on real RAG products actually shows

The clearest data point here isn't theoretical — it's a preregistered empirical study out of Stanford's RegLab that tested the two leading RAG-based legal research products (LexisNexis's Lexis+ AI and Thomson Reuters's Westlaw AI-Assisted Research), both marketed at the time as avoiding hallucination. The result: both still hallucinated in 17% to 33% of tested queries — a real improvement over general-purpose models answering from memory alone (which the same research group found hallucinating on the majority of comparable legal questions), but nowhere near zero (Stanford RegLab, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools").

The mechanism behind that gap is well understood, and it splits into two places things go wrong:

• Retrieval failure — the search step doesn't surface the passage that actually answers the question, either because the query was ambiguous, the right document wasn't indexed well, or a plainly relevant chunk got ranked below less-relevant ones.
• Generation failure — the model has the right passage in front of it, and still produces a claim that passage doesn't actually support: overgeneralizing, blending it with something from its own training data, or getting confused when multiple retrieved passages disagree (sometimes called the "lost in the middle" effect, where a model under-weights information buried in a long context).

Neither failure mode is exotic or rare enough to ignore. Grounding an answer in retrieved documents makes hallucination less likely, measurably — it does not make it structurally impossible.

What actually helps, beyond retrieval alone

Given that framing, the useful engineering question isn't "did we add RAG" — it's "what happens after retrieval to catch the two failure modes above." A few concrete levers matter:

• Better retrieval, not just retrieval. Combining vector similarity search (which finds conceptually related text) with keyword search (which catches exact terms vector search can miss) and fusing the two — a common approach is Reciprocal Rank Fusion — then reranking the merged results, catches more of the retrieval-failure case than either method alone.
• Checking the output against the source, after generation. Rather than trusting that a grounded prompt produced a grounded answer, a separate pass can check each substantive claim in the draft answer against the specific passage it's supposedly drawn from, and force a re-retrieval when a claim isn't actually supported. This is the piece that catches the generation-failure case specifically — the one where retrieval did its job and the model still drifted.

This is the shape of what AssistantGeneral does with its own retrieval pipeline: hybrid retrieval fuses vector and keyword search with Reciprocal Rank Fusion, then reranks and deduplicates the results, and a groundedness gate checks every substantive claim in a draft answer against a retrieved source before it reaches the user, triggering a closed re-retrieval pass on anything unsupported. Separately, a knowledge graph links documents by the topics and keywords they share, so retrieval can walk to a related source even when the first search misses it.

None of that is a claim that hallucination is solved — the research above is exactly why we wouldn't put that sentence on a product page. It's an attempt at closing the two specific gaps that plain RAG leaves open: bad retrieval, and generation that drifts from what was actually retrieved. Whether it's working is ultimately a question of measured accuracy on your own documents, not a property you can claim once and stop checking.