Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference

If you set temperature to zero, a language model should pick the most likely token every time and produce the same answer every time. Evaluations rely on this: they run greedy decoding once and report the score. This paper shows that the assumption breaks as soon as the serving setup changes. With the same prompt and seed, changing the evaluation batch size, the number of GPUs or the GPU model changed the outputs, and for reasoning models it changed the accuracy.

The cause is ordinary floating-point arithmetic. Addition in finite precision is not associative: (a + b) + c can differ from a + (b + c). Matrix multiplications and reductions on GPUs add numbers in orders that depend on the batch size, how work is split across GPUs, and the kernel the hardware picks. In bfloat16, which keeps only 8 bits of precision, those small differences are large enough to change a logit. When the top two tokens are close, a tiny change flips the greedy choice, and from that token on, the two runs are generating different text. Reasoning models, which write thousands of tokens of chain of thought, give that flip thousands of chances to happen.

The authors measured it across 12 configurations: two GPU types (NVIDIA L40S and A100), two GPU counts and three batch sizes, in bfloat16, float16 and float32. For DeepSeek-R1-Distill-Qwen-7B on AIME'24, accuracy had a standard deviation of 9.15% in bfloat16, 5.74% in float16 and zero in float32. Output length varied with a standard deviation of over 9,000 tokens in bfloat16. Non-reasoning models varied far less.

Float32 everywhere fixes it but doubles memory and time. The proposed LayerCast stores linear-layer weights in bfloat16 and casts each one to float32 just before using it, so all arithmetic is float32. It matched float32's determinism and used 34% less memory than full float32 inference.

Every FDE eventually hears "we set temperature to zero, so why did the answer change?" This paper is the explanation and the guide to what to do about it.

Your evaluation numbers depend on your serving setup. An eval run on 1 GPU at batch 8 and a production deployment on 4 GPUs at batch 64 can disagree, even with greedy decoding. When comparing a customer's numbers with a vendor's or a paper's, record precision, GPU type and count, and batch size alongside the score.

Single runs are not enough for reasoning models. A one-run accuracy on a 30-question benchmark like AIME'24 can move by several questions just from configuration. Report the mean and spread over several configurations or seeds, and treat small differences between models as noise.

Flaky regression tests have a root cause. If an LLM regression test that compares outputs exactly fails intermittently, numeric nondeterminism is a likely cause. Either pin the configuration and use float32 for the test, or compare outputs semantically instead of byte by byte.