TL;DR: Generation is an autoregressive loop of single-token predictions, split operationally into prefill (process the prompt in parallel, build the KV cache, sets time-to-first-token) and decode (emit output tokens one at a time, sets total latency).
How to approach it
Picture the answer that loses the round: "the model looks up the closest match in its training data and returns it." Nothing in the machinery looks anything up, and the rest of your answer should make that sentence impossible to believe. This is a calibration question: the interviewer wants to know whether you have a working mental model or just vibes. Answer at the systems level (what happens between request and response) rather than at the level of attention-head math. A good move is to say "I'll describe the inference loop, and I'm happy to go deeper on the architecture if useful."
A strong answer
An LLM is a next-token predictor. Your prompt is split into tokens, and the model computes a probability distribution over its entire vocabulary (roughly 50k to 200k tokens) for what comes next. A sampling strategy (temperature, top-p) picks one token, it gets appended to the sequence, and the process repeats one token at a time until a stop condition. Generation is an autoregressive loop over single-token predictions.
Make that concrete with one step. Feed the model "The capital of France is" and the forward pass produces a score for every token in the vocabulary. After softmax, the distribution might look like (illustrative numbers): " Paris" 0.87, " the" 0.04, " located" 0.02, and a long tail of thousands of tokens sharing the remainder. At temperature 0 you take " Paris" every time. At temperature 1 you sample, so once in a while you get " the", and the sentence heads somewhere else entirely. Now run that loop three hundred times and you have a paragraph. Nothing in the machinery ever plans the paragraph; each step only ever chose one plausible next token. If you have used a phone keyboard's word suggestions, you have seen the same mechanism, just with a model too small to hold a conversation. The difference is scale and context, not kind.
Two phases matter operationally. Prefill processes the whole prompt in parallel to build the KV cache; this dominates time-to-first-token and scales with input length. Decode then produces output tokens serially, typically around 30 to 150 tokens/second depending on model size and serving stack; this dominates total latency and scales with output length. That split is why a 5,000-token answer is slow no matter how short the prompt, and why streaming exists: you show tokens as decode produces them.
The dashed arrow running down the left is the whole latency story. It fires once per output token, so the number of trips around it is the length of the answer, and nothing about the prompt changes how many trips there are. Stages 3 to 8 happen once for the prompt, in parallel; every token after the first pays for the whole stack again on its own.
This model explains the behaviors customers ask FDEs about. The model isn't retrieving facts from a database; it continues text in the statistically most plausible way, which is why fluent, confident, wrong output (hallucination) is a natural failure mode rather than a bug. It completes with total confidence whether or not the truth cooperates. It also explains why the same prompt can yield different answers at temperature > 0, and why "did the model read my whole document?" is really a question about what made it into the context window.
The two-phase split also explains a thing every customer eventually asks about: the bill. Look at any provider's price sheet and output tokens cost several times more than input tokens. That is not marketing, it is the mechanics you just described. Prefill processes the whole prompt in one parallel pass, so a thousand input tokens share the cost of roughly one trip through the model. Decode pays a full forward pass per token, serially, holding the GPU the entire time. When a customer wants their LLM bill cut, this is why "make the answers shorter" is usually worth more than "make the prompts shorter", and being able to derive that from first principles in the room is exactly what separates an FDE from someone reading the pricing page aloud.
One sentence on training: pretraining on internet-scale text teaches the next-token objective; post-training (instruction tuning, RLHF) shapes it into a helpful assistant. The generation mechanics are identical for both.
What interviewers probe next
- "So why does the same prompt give different answers?" Sampling at temperature > 0; set temperature 0 or fixed seeds for near-determinism, but note serving-side nondeterminism (batching, floating point) can still cause tiny variations.
- "What drives latency, input or output tokens?" TTFT scales with input (prefill); total time scales mostly with output (serial decode). So trimming output length usually buys more than trimming the prompt.
- "Does the model know when it's wrong?" No explicit confidence; token probabilities exist but are poorly calibrated for factual truth, which motivates external verification and evals.
- "What is the KV cache actually for?" Without it, every new token would recompute attention over the entire sequence from scratch, making step N cost N times the work. Caching each token's keys and values makes decode incremental: one forward pass per new token. The cost is memory that grows with sequence length, which is why long chats eventually pressure serving capacity.
- "Where would you batch?" Decode is memory-bandwidth-bound, so serving many requests in one batch reuses each weight read across all of them. That is the single biggest throughput lever in serving, and volunteering it turns a fundamentals question into a systems conversation you control.
The answers that sound right and lose, against what the mechanism actually says:
| What people reach for | Why it fails | What to say instead |
|---|---|---|
| "It looks up the answer in its training data" | Nothing is stored as an answer or searched; the forward pass produces a distribution over the next token | "It samples one token at a time from a distribution the forward pass computes" |
| "Latency depends on how long the prompt is" | Prompt length sets time-to-first-token only; total time follows output length because decode is serial | "Prefill is parallel, decode is one forward pass per token" |
| "Temperature 0 makes it deterministic" | Batching and floating-point order on the server still perturb the logits | "Near-deterministic; serving nondeterminism remains" |
| "Hallucination is a bug" | The objective is plausible continuation; a fluent fabrication is that objective succeeding | "It is the objective working, so we bound it with grounding and evals" |
Common mistakes
- Describing the model as "looking up" answers or "searching its training data": instant credibility loss.
- Jumping into transformer internals (Q/K/V matrices) when the question is operational: wrong altitude for an FDE round.
- Not knowing that output tokens are generated serially. This is the root of most latency conversations you'll have with customers, and interviewers check for it.
- Claiming temperature 0 makes output fully deterministic across all providers; savvy interviewers know about serving nondeterminism.
Key takeaways
- Generation is one repeated operation: predict a distribution over the vocabulary, sample one token, append, repeat.
- Prefill sets TTFT and scales with input length; decode sets total latency and scales with output length.
- Hallucination falls out of the objective (plausible continuation, not verified truth), so it is expected, not a defect.
- In the room: volunteer the prefill/decode split before it is asked, then price a customer's bill from it. That one move turns a warm-up into a serving conversation you control.