I built RAG and it still hallucinates — why?

Usually retrieval is at fault: the right passage never arrives and the model composes an answer from irrelevant text. Measure retrieval and generation separately. If the right passage did arrive and the answer is still wrong, the problem is in the prompt — the 'rely only on the supplied text' instruction may be missing.

How do I measure the hallucination rate?

With around a hundred questions whose correct answers and correct sources are written by hand. For each answer, ask two things: is the claim present in the source, and was the source the right document. Track those two rates separately; looked at together, you cannot see which one broke.

Does telling the model to flag uncertainty help?

Partly. Self-reported confidence correlates weakly with actual accuracy — models are wrong while sounding certain. It is useful but not sufficient, and it does not replace source checking or human approval.

← All notes

Accuracy

Why AI makes things up

A hallucination is a language model stating something untrue with confidence. The cause is not a fault but the design: the model produces the most fitting continuation, not the true one. It therefore cannot be eliminated — but with the right setup it can be reduced enough for a system to be usable.

Published: 2026-09-06
Built by: Zero-Shot Team
4 min read

The model is not lying, it is predicting

A language model computes which word is likely to follow. There is no layer that asks 'is this true' — only probability. Asked for a product code, a legal clause or a date it does not know, the model does not notice that it does not know; it produces something shaped to fit the gap.

That is why hallucinations cluster exactly where factual precision matters: numbers, names, dates, citations. And those are precisely the places where a wrong answer costs the most.

The confident tone has the same origin: text is generated to be fluent, and confidence is not reflected in the wording. Where a person would say 'I think', the model writes a flat statement.

The most effective fix is giving the model the facts

Rather than asking about something the model does not know and hoping, put the text containing the answer into the prompt. That is exactly what enterprise retrieval (RAG) does, and it reduces hallucination substantially.

But it is not sufficient. When the wrong passage is retrieved, the model summarises that wrong passage fluently and the result is more convincing, not less. So retrieval accuracy must be measured separately.

The second condition is instructing the model to rely only on the supplied text, and to decline when the text does not contain the answer. Without that sentence, the model fills gaps from its own memory.

Citations are an accuracy mechanism

Showing which document each claim came from does two things. The user can verify, and the model constrains itself — an answer that must carry a citation contains less invention.

The citation must be clickable and must actually contain the fact. A fake citation is worse than none: it creates trust that does not survive checking.

A practical control: automatically check whether every number and proper noun in the answer appears in the source. If not, flag the answer.

What does not work

Telling the model 'don't make things up' does little on its own. It cannot stop doing what it does not know it is doing.

A bigger model does not solve it either. Larger models hallucinate less but more persuasively; while retrieval is broken, scaling the model raises cost, not accuracy.

Setting temperature to zero makes answers consistent, not correct. It gives you the same wrong answer every time.

Managing the residual risk

No system can promise zero hallucination. What can be built is a flow where errors are catchable.

Three things are needed: human approval where a wrong answer is costly, a citation beside every answer, and a way for users to report errors. Every reported error joins the evaluation set, so the same failure does not recur.

And honesty: the system's limits should be stated to users in writing. 'This assistant can be wrong — check the source for important decisions' does not reduce trust; it prevents trust from collapsing the first time it is.

In short

Hallucination is a consequence of how models work, not a defect; it is reduced, not removed.
The strongest fix is supplying the answering text — but retrieval accuracy must be measured on its own.
'Rely only on the supplied text; decline if it isn't there' belongs in the prompt.
Show clickable citations, and automatically verify that numbers and names appear in the source.
Bigger models and temperature settings are not fixes; keep human approval where errors are costly.

Common questions

I built RAG and it still hallucinates — why?: Usually retrieval is at fault: the right passage never arrives and the model composes an answer from irrelevant text. Measure retrieval and generation separately. If the right passage did arrive and the answer is still wrong, the problem is in the prompt — the 'rely only on the supplied text' instruction may be missing.
How do I measure the hallucination rate?: With around a hundred questions whose correct answers and correct sources are written by hand. For each answer, ask two things: is the claim present in the source, and was the source the right document. Track those two rates separately; looked at together, you cannot see which one broke.
Does telling the model to flag uncertainty help?: Partly. Self-reported confidence correlates weakly with actual accuracy — models are wrong while sounding certain. It is useful but not sufficient, and it does not replace source checking or human approval.

More notes

All notes →