TL;DR: Hallucination is the training objective working as designed: the model produces plausible continuations, not verified truth, and RLHF rewards sounding confident. You can't prompt it away, so you engineer around it with grounding, citations, permission to abstain, and groundedness evals.
How to approach it
The wrong move is to defend the model: "it gets confused sometimes, the next version will be better." The customer hears a promise you cannot keep. Open with the mechanism instead, because a mechanism can be bounded and measured and a mood cannot. This question tests whether you can give a mechanistic explanation rather than a hand-wavy one ("the model makes things up sometimes"). Structure: the training objective, why that objective produces confident fabrication, the aggravating factors, and one sentence on what this implies for deployment. Anthropic's loop famously includes a customer-scenario version ("customer says Claude is hallucinating, walk me through it"), so practice both registers.
A strong answer
Hallucination is the natural consequence of the training objective. The model is trained to produce plausible continuations (text that is statistically likely given the context) rather than verified true statements. There is no internal database lookup and no truth oracle; when the model "answers," it is generating the most probable-sounding sequence. When the truth is well-represented in training data, plausible and true coincide. When it is not, the model interpolates, and fluent fabrication is exactly what a plausibility machine produces.
Notice that "Confident fabrication" has two arrows into it in the flowchart. The first is the retrieval miss everyone expects. The second comes from "grounded but unfaithful", and that path is why RAG lowers the rate without ever taking the box off the diagram.
It helps to watch the objective fabricate one artifact token by token. Ask for supporting case law and the model begins a citation: a party name is a fine continuation, then "v.", then another surname, then the reporter format it has seen hundreds of thousands of times, a volume number in the right range, a plausible page, a circuit and year that are mutually consistent. Judge each token by the only standard the model was ever trained on, "is this a likely next token here?", and every single one is an excellent prediction. The citation as a whole was assembled, not retrieved; whether it happens to match a real case is chance. This is why "the model made a mistake" is the wrong mental model: no individual step was a mistake by the objective's own lights. Fabrications are not low-confidence output leaking through a broken filter. They are the machine performing well on a different problem than the one the user thinks it is solving.
Several factors aggravate it:
- Knowledge gaps with no abstention instinct. Pretraining does not teach the model to distinguish "I've seen this fact thousands of times" from "I'm pattern-matching from adjacent material." The output looks identical either way.
- Post-training rewards confidence. RLHF optimizes for responses humans prefer, and humans prefer fluent, complete, confident answers over hedging. The training process can actively reward sounding sure. Kalai et al. (2025, 'Why Language Models Hallucinate') put the mechanism plainly: models hallucinate because training and evaluation reward guessing over acknowledging uncertainty, and the errors arise as ordinary binary-classification mistakes whenever incorrect statements cannot be told apart from facts. Their sharper point is about benchmarks: most are graded so that guessing when uncertain improves the score, so a model optimized to be a good test-taker learns to guess.
- Leading or ambiguous prompts. Ask "Summarize the 2022 Smith v. Acme ruling" and the model will oblige even if no such case exists, because the prompt presupposes the fact.
- Long or conflicting context. Retrieved chunks that partially contradict each other, or facts buried mid-context, push the model back onto its priors.
- Specifics decay fastest. Citations, URLs, numbers, and names are high-precision targets where "approximately right" is flatly wrong, which is why fabricated case law and fake references are the canonical examples.
The deployment implication, stated unprompted: you can't prompt hallucination away; you engineer around it. Ground the model in retrieved sources with citations, constrain what it is allowed to assert, give it explicit permission to say "I don't know," and build evals that measure groundedness against your corpus. The customer-facing framing I use: the model is not lying or broken; it is doing exactly what it was trained to do, and our job is to bound where "plausible" is allowed to diverge from "true."
When this question arrives as a live escalation instead of an interview prompt, the explanation has to become a procedure, and describing that procedure is what makes the answer sound like deployment experience. Take the customer's worst example into the meeting, not a defense of the model. Run the one diagnostic that splits the problem: was the correct information in the context the model saw? If it never got there, this is a retrieval or ingestion failure and the flowchart's "grounded?" branch is where the work is; if it was there and the model contradicted it, that is a faithfulness failure and the fixes are citation requirements, tighter grounding instructions, and abstention. Then leave the room with a number and a bound rather than a promise: "we measured groundedness on 100 of your real queries, here is the rate, here is the mode that refuses to answer without a citation, and here is how the rate trends per release." Customers de-escalate on evidence of control, not on assurances that the model is improving.
What interviewers probe next
- "Does temperature 0 fix it?" No; it removes sampling variance, not wrong priors. A confidently wrong model is wrong deterministically.
- "Does RAG eliminate it?" It reduces it, does not eliminate it: retrieval can miss, and models can ignore or distort provided context (faithfulness failures). You still need groundedness evals.
- "How would you measure hallucination rate?" Build an eval set from real queries with sourced answers; score groundedness with an LLM judge calibrated against human labels; track it per release.
- "Will bigger models stop hallucinating?" Rates improve with scale and better post-training, but the objective has not changed; plan deployments assuming a nonzero rate.
What people say about it, and what the objective says back:
| What people reach for | Why it fails | What to say instead |
|---|---|---|
| "The model lies" or "gets confused" | There is no belief to betray, only a distribution to sample from | "It continues text plausibly; where plausible and true diverge, you get fabrication" |
| "Temperature 0 fixes it" | Removes sampling variance, not wrong priors; the wrong answer becomes reliable | "Deterministic and wrong is still wrong" |
| "RAG eliminates it" | Retrieval can miss, and the model can ignore or distort what it was handed | "RAG reduces it; groundedness evals measure what is left" |
| "Bigger models will stop doing this" | Rates fall with scale and better post-training; the objective has not changed | "Plan for a nonzero rate and bound it" |
Common mistakes
- "The model lies or gets confused": anthropomorphizing instead of explaining the objective.
- Promising any complete fix via RAG, fine-tuning, or prompts. Interviewers immediately distrust candidates who claim zero hallucinations is achievable.
- Missing the RLHF-confidence connection. It is the detail that separates a mechanistic answer from a recited one.
- Giving only theory. An FDE answer must land on "and therefore, in production, we..."
What to actually do
Take the customer's worst example into the meeting, not a defense of the model. Check whether the correct information was in the context the model saw: if it was not, fix retrieval or ingestion; if it was, tighten grounding, require citations, and give the model explicit permission to abstain. Measure groundedness on 100 of the customer's real queries and report the rate per release. Leave the room with a number and a bound, never with a promise.
Key takeaways
- Hallucination is the objective behaving correctly (plausible continuation), so it is expected, not a bug.
- RLHF rewards confident phrasing, which is why fabrications arrive fluent rather than hedged.
- You bound it, not banish it: grounding plus citations plus permission to abstain plus groundedness evals, assuming a nonzero rate.