Why LLMs Hallucinate

TL;DR: A model predicts plausible text, not true text. It has no internal fact-checker and no concept of "I don't know" unless you build one, so when it lacks the answer it fills the gap with something that sounds right. You reduce this by grounding the model in retrieved sources, instructing it to refuse when unsupported, demanding citations, and measuring faithfulness with evals. You do not eliminate it.

Why it happens at all

The demo goes well until someone asks a question the documents do not cover, and the assistant answers anyway, in the same confident voice it used when it was right. That moment is the whole topic.

A language model is trained to predict the next token that best continues the text, judged against enormous amounts of training data. That is the entire objective. Truth is not in the loss function. The model learns what plausible text looks like, and a confident, specific, well-formed answer looks plausible whether or not it is correct.

So when you ask for a fact the model never saw, or saw rarely, or saw contradicted across sources, it does not have a "no data here" signal to return. It produces the most likely continuation, which is often a fluent, authoritative-sounding fabrication: a citation to a paper that does not exist, an API method that was never built, a refund policy invented because thirty days is common. The fluency is the trap. The same machinery that makes correct answers sound confident makes wrong ones sound confident too.

This is why hallucination is not a bug to be patched out. It is a direct consequence of how the model works. A model with no access to ground truth and an objective of "sound plausible" will sometimes be plausibly wrong.

How to explain it to a non-technical exec

You will be asked this in a room with a buyer. The line that lands: "The model is an extremely well-read improviser, not a database. Ask it something it knows well and it is reliable. Ask it something outside what it learned and it will still answer confidently, because guessing fluently is what it was trained to do. Our job is to give it the right documents to answer from and to make it say 'I don't know' when it has none." That reframes the problem from "the AI lies" to "we control what it is allowed to answer from," which is what makes the deal safe.

Mitigations that actually work

Before the mitigations, the spine above traces where a fabrication comes from, because every one of those mitigations attacks a specific step in it.

• Grounding via RAG. Retrieve the relevant documents and instruct the model to answer only from them. This swaps "recall from fuzzy memory" for "read from a source," which is far more reliable.
• Refuse when unsupported. The single most valuable instruction in a grounded system: "If the context does not contain the answer, say you do not know." A model that abstains beats one that guesses, especially in regulated or high-stakes settings.
• Citations. Require the model to point at the source passage for each claim. Citations let a human verify and make fabrication visible, because a made-up claim has no real passage to point to.
• Evaluation. Measure faithfulness (does the answer follow from the provided context) on a golden set, often with an LLM-as-a-judge (a second model prompted to grade whether each claim in the answer is supported by the context) plus human spot-checks. You cannot manage what you do not measure, and a faithfulness metric is what tells you a prompt change helped or hurt.

None of these makes the model "know" truth. They constrain it to answer from material you control and surface the cases where it cannot.

Why interviewers probe this

It is the question behind every enterprise objection, so they want to see that you neither overpromise nor panic. The answer that fails is "we'll fine-tune it to stop hallucinating," which misunderstands the cause; fine-tuning changes behavior and style, not the model's access to current truth. The follow-up they hold in reserve is "your RAG system still hallucinated on a question where the right document was retrieved, why?" Strong answer: the model ignored or misread the context, so you tighten the refuse-when-unsupported instruction, lower temperature, and check whether the retrieved chunk actually contained the answer or just looked relevant.

Common misconceptions

• "Hallucination means the model is broken or low quality." It is inherent to predicting plausible text without grounding; even frontier models do it.
• "RAG eliminates hallucination." It reduces it by grounding, but the model can still ignore or misread the retrieved context. You still need refusal and faithfulness checks.
• "Fine-tuning will fix factual errors." Fine-tuning shapes behavior and format and bakes in static facts at best; it does not give the model live, citable truth. Use grounding for facts.
• "Higher temperature is the cause." Sampling can surface unlikely tokens, but the root cause is the lack of grounding and any notion of truth, not the sampler.

Key takeaways

• Models predict plausible next tokens with no built-in concept of truth, so confident fabrication is expected behavior, not a defect.
• Mitigate with grounding (RAG), an explicit instruction to refuse when unsupported, required citations, and faithfulness evals.
• Explain it to non-technical stakeholders as "a well-read improviser, not a database," and emphasize that you control what it answers from.
• You manage hallucination; you do not eliminate it. Fine-tuning changes behavior, not access to current truth.