TL;DR: Temperature rescales the next-token distribution (0 is greedy argmax, higher flattens it); top-p keeps the smallest set of tokens whose probability sums past p. Run temperature 0 for extraction, structured output, tool calls, and evals, but say plainly that it cuts variance, not hallucination.
How to approach it
Define both parameters crisply, state the practical defaults, then answer the real question: which enterprise workflows want determinism and which want diversity. Flag the common misconception (that temperature 0 means "no hallucinations") before the interviewer has to.
A strong answer
At each step the model outputs a probability distribution over the next token. Temperature rescales that distribution before sampling: low temperature sharpens it toward the most likely tokens; high temperature flattens it, giving lower-probability tokens more chance. At temperature 0 you effectively take the argmax every time, which is greedy decoding. Temperature 0 is not a truth switch; it is the same guess, made consistently. Top-p (nucleus sampling) instead truncates the distribution to the smallest set of tokens whose cumulative probability exceeds p (for example 0.9) and samples within it; it adapts how many candidates survive based on how confident the model is. They interact, so the standard advice is to tune one and leave the other at default.
Numbers make both knobs concrete. Suppose the model's next-token distribution over four candidates is 0.60, 0.25, 0.10, 0.05 at temperature 1. Rescaling changes those to:
| Temperature | Token A | Token B | Token C | Token D | Effect |
|---|---|---|---|---|---|
| 0.5 | 0.83 | 0.14 | 0.02 | 0.01 | Sharpened: the favorite dominates |
| 1.0 | 0.60 | 0.25 | 0.10 | 0.05 | The model's raw distribution |
| 2.0 | 0.43 | 0.28 | 0.17 | 0.12 | Flattened: underdogs get real chances |
| 0 | 1.00 | 0 | 0 | 0 | Greedy: always token A |
On the same distribution, top-p 0.9 keeps tokens A, B and C (cumulative 0.60, 0.85, 0.95: the set crosses 0.9 at the third token) and never samples D. Notice what top-p does that a fixed top-k cannot: when the model is confident and one token holds 0.95 alone, top-p keeps just that token; when the distribution is flat across forty plausible tokens, it keeps forty. The candidate pool adapts to the model's own certainty, which is why top-p aged better than top-k as the default.
When I'd run temperature 0 (or near it) in enterprise workflows:
- Extraction and classification: pulling fields from invoices, routing tickets, flagging clauses. You want the same input to give the same output.
- Structured output: JSON for downstream systems, where sampling creativity only adds schema breakage.
- Tool/function calling: argument generation should be boring and repeatable.
- Evals and debugging: reproducibility makes regressions measurable; running evals at high temperature adds noise you then misread as quality change.
Where temperature > 0 earns its keep: marketing copy variants, brainstorming, synthetic data generation (you want diversity), and conversational tone where slight variation feels natural. Typical values: 0 to 0.3 for deterministic tasks, around 0.7 for general chat, 1.0 and up for creative generation.
Two caveats that signal seniority. First, temperature 0 reduces variance, not error; a model that is confidently wrong is wrong at every temperature, because hallucination is a grounding problem, not a sampling problem. Second, temperature 0 is not perfectly deterministic in practice: batched serving and floating-point nondeterminism can still flip a token occasionally, so don't promise customers bit-identical outputs. Promise low variance plus eval-backed quality.
What interviewers probe next
- "Customer set temperature 0 and still gets hallucinations, what do you tell them?" Sampling controls variance; grounding (RAG, citations), better prompts, and verification control correctness.
- "When is temperature 0 actively harmful?" Diversity tasks; also greedy decoding can produce repetitive or degenerate text in long free-form generation.
- "How do you choose the value?" Empirically: build a small eval set and sweep; don't cargo-cult 0.7.
Common mistakes
- Hand-waving the mechanics ("temperature is creativity") with no distributional explanation. Fine for customers, too shallow for an AI-lab interview.
- Claiming temperature 0 guarantees determinism or factual accuracy. Both wrong, and interviewers specifically listen for these.
- Recommending tuning temperature and top-p simultaneously without acknowledging they interact.
- Not connecting to evals. Choosing sampling params by vibes instead of measurement is exactly the habit FDE interviews screen against.
Key takeaways
- Temperature 0 is greedy argmax; it lowers variance but not error, so it never fixes hallucination.
- Tune one of temperature or top-p and leave the other at default, because they interact.
- Greedy is right for extraction, JSON, tool calls, and evals; diversity tasks want temperature > 0.