Why Language Models Hallucinate

A model that earns a point for a correct answer and loses nothing for a wrong one has a reason to guess. Saying it does not know earns nothing too. Even a poorly supported answer offers some chance of increasing its score. That incentive is a central explanation in this paper for why confident factual errors survive post-training.

The other half concerns pretraining. The authors connect generating valid text to distinguishing valid statements from errors. If the available data and model cannot reliably make that distinction, learning the language distribution can assign probability to plausible falsehoods even when the training examples themselves are correct. Rare arbitrary facts are a useful example: knowing the birthdays of many people does not reveal another person's birthday.

The mathematical result has assumptions and correction terms. It is not a proof that every assistant must make factual errors, nor a measured hallucination rate for your system. A restricted application can answer from verified records and abstain elsewhere. The paper explicitly recognizes that possibility.

For deployment, its most useful recommendation is to change what success means. Score appropriate abstention and costly errors separately, and tell the system which confidence target matters. Then measure whether the resulting behavior matches that target.

A customer asking for fewer hallucinations usually wants fewer unsupported answers reaching a consequential workflow. An aggregate accuracy score cannot tell you whether an updated assistant became safer or simply answered more questions.

Consider an assistant that explains account-specific policy. A wrong answer may trigger rework; an explicit unknown can go to an operator with the relevant record attached. These outcomes have different costs. Your evaluation should preserve that difference instead of marking both as equally unsuccessful.

Start by specifying what counts as supported, when the assistant may request clarification, and who receives an unresolved case. Compare the fraction of requests answered with the error rate among those answers. Review both on difficult customer slices. The calibration and uncertainty concept provides the measurement vocabulary; this paper explains why the incentives behind that measurement matter.