The Internal State of an LLM Knows When It's Lying

Language models generate one token at a time and commit to each one. The authors argue that this can lead a model into a false statement even when it has the information to recognize it as false afterwards. Their example: having written "Pluto is the smallest", a model is pushed toward "dwarf planet in our solar system", which is wrong. If the model's internal state represents truth or falsehood, that representation could be read out.

They build a dataset of 6,084 true and false statements across six topics: cities, inventions, chemical elements, animals, companies and scientific facts. For the first five, true statements come from reference tables, and false ones swap in a property from a different row ("The atomic number of Hydrogen is 34"). Scientific facts and their negations come from ChatGPT, checked by two annotators.

The method, SAPLMA (Statement Accuracy Prediction based on Language Model Activations), feeds each statement to the model, records the 4,096 activations of one hidden layer at the last token, and trains a small feed-forward network (256, 128 and 64 units) to output the probability the statement is true. Training uses five topics; testing uses the sixth. The authors try five layers in OPT-6.7b and LLaMA2-7b.

Averaged over held-out topics, the best layer reached 71% with OPT-6.7b and 83% with LLaMA2-7b. Few-shot prompting of OPT-6.7b and a classifier on BERT embeddings stayed near 54%. The authors then test on 245 statements that OPT-6.7b generated itself, labeled by three people. There the probe reached 62% to 64% at the default threshold and 71% when the threshold was tuned on 30% of the statements. GPT-4, given the same statements and asked to label them, reached 84.4%.

The signal exists. Hidden activations carry information about truth that a small classifier can pick up, even across topics it was not trained on. This is the starting point for a family of white-box hallucination detectors, and the idea of probing internal states has stayed influential.

The deployment case is the weak one. The headline 71% to 83% comes from template statements like "Oranjestad is a city in Aruba". On statements the model actually generated, the probe reached 64% at its default threshold and 71% only after tuning on part of the test set. A customer evaluating a "hallucination detector" built on this idea should ask for numbers on their own generated outputs.

A prompted checker may beat it. On the same generated statements, GPT-4 asked to label each one reached 84.4%. A probe needs access to model internals, which closed APIs do not give, and a training pipeline. A second model asked to check claims needs neither. Compare both on the customer's data before building either.

Thresholds need calibration. The probe called about 70% of generated statements true at the default 0.5 threshold, and the tuned thresholds were 0.81 to 0.91. Any confidence score from a detector has to be calibrated on data that looks like production before you set an action threshold, and the authors make this point themselves.

Use it as a flag, not a verdict. At 70% accuracy, a probe can prioritize statements for review or for retrieval-based checking. It cannot be the only gate on a high-stakes claim.