Temperature, Top-p and Sampling

TL;DR: A model does not pick the next word; it produces a probability for every possible next token, and sampling decides how to choose from that distribution. Temperature reshapes the distribution to be flatter or sharper, top-p and top-k cut off the unlikely tail, and greedy decoding just takes the most probable token. Use low settings when you want one correct answer, higher when you want variety.

What the model actually outputs

At every step, the model computes a score (a logit) for each token in its vocabulary, then converts those scores into a probability distribution that sums to 1. The token "Paris" might get 0.6, "London" 0.2, the rest split among thousands of others. Decoding is the rule for turning that distribution into a single chosen token. The model never "decides"; the sampling strategy does.

The knobs

• Greedy decoding. Always take the highest-probability token. Deterministic and repeatable, but can be flat and repetitive: it commits to the single top token every step, so a confidently wrong token gets picked with no diversity to escape it.
• Temperature. The model's raw scores (logits) are divided by the temperature before the softmax turns them into probabilities. Temperature below 1 sharpens the distribution, making likely tokens even more likely (more deterministic). Above 1 flattens it, giving long-shot tokens a real chance (more random). At 0 you effectively get greedy decoding.
• Top-k. Keep only the k most probable tokens and sample from those, discarding the rest. Caps how adventurous the choice can get.
• Top-p (nucleus sampling). Keep the smallest set of tokens whose probabilities add up to p (say 0.9), then sample from that set. Unlike top-k, the set size adapts: when the model is confident, the nucleus is tiny; when it is unsure, the nucleus widens.

The order is not cosmetic. Temperature is applied to the raw scores before the softmax, so it reshapes the distribution that top-k and top-p then cut, and swapping those two steps gives a different answer.

Drag the temperature below to watch the same logits reshape into a peaked or a flat distribution:

When to set what

The setting follows the task. For anything with one right answer, classification, extraction, structured JSON, code that must parse, function-call arguments, drive temperature low (near 0) and lean on greedy or tight top-p. You want the same input to give the same output and no creative detours.

For ideation, marketing copy, brainstorming alternatives, or any case where you want several different drafts, raise temperature and widen top-p so the model explores. A typical move is to generate at moderate temperature and sample several candidates, then pick or rerank.

The mistake is treating temperature as a quality dial. Higher temperature is not "smarter," it is more random; lower is not "dumber," it is more focused. You are trading variety against reliability, not turning intelligence up or down.

Why "different every run" is not a bug

The first time someone runs the same prompt twice and gets two different answers, they file a bug. It is not one. Any sampling with temperature above 0 is stochastic by design; you asked the system to roll weighted dice. If you need reproducibility, set temperature to 0 (and where the API offers it, a fixed seed). Even then, expect occasional drift across model versions or infrastructure, so true bit-for-bit determinism is not guaranteed and you should not architect around it.

Why interviewers probe this

It separates people who have only chatted with a model from people who have called the API for a product. The tell is someone who says "set temperature higher to make it more accurate," which is backwards. The follow-up they hold in reserve: "your JSON extractor occasionally returns malformed output, what do you check first?" A strong answer starts with "is temperature above 0, and am I constraining the output format," before touching the prompt.

Common misconceptions

• "Temperature controls quality." It controls randomness. For factual tasks, lower is more reliable; that is not the same as smarter.
• "Top-k and top-p do the same thing." Top-k fixes the number of candidates; top-p fixes their cumulative probability, so its set size adapts to the model's confidence.
• "Temperature 0 guarantees identical output forever." It removes sampling randomness, but model or infrastructure changes can still cause drift; do not assume perfect reproducibility.
• "Sampling randomness causes hallucination." It can surface low-probability tokens, but the root cause of hallucination is the model predicting plausible text without grounding, not the sampler.

Key takeaways

• The model outputs a probability over all next tokens; sampling chooses one, so decoding settings, not the model alone, determine the output.
• Temperature reshapes the distribution (sharper below 1, flatter above); top-k and top-p truncate the unlikely tail.
• Use near-0 temperature for extraction, classification, and structured output; raise it for brainstorming and variety.
• Different answers across runs at temperature above 0 are expected behavior; set temperature to 0 when you need determinism.