TL;DR: Trace five stages (tokenize/embed, position + residual stream, attention, feed-forward/MoE, next-token sampling) and keep one distinction sharp: attention routes information between tokens, the feed-forward network stores the world knowledge.
How to approach it
The wrong move here is fluent. A candidate explains Mixture of Experts as "a smaller, cheaper model" and locates the model's knowledge "in the attention weights." Both sound right, both are backwards, and both are exactly what the interviewer is listening for. Set the altitude before you start: you are giving the architectural skeleton, not deriving backpropagation. Say which path you will trace (how text becomes output) and then walk it: tokens, embeddings, position, attention, the feed-forward block, and sampling. What the interviewer is really checking is whether you can hold the whole pipeline in your head and whether you know which part handles communication between tokens and which part stores knowledge. Most candidates blur those two; keeping them separate is the tell of someone who has actually read the internals.
A strong answer
Every modern model shares the same five stages.
Two boxes in the diagram do all the work the interviewer cares about, stages 5 and 6, and the dashed arrow between them is one layer repeating. Everything above them prepares their input and everything below reads their output.
-
Tokenization and embeddings. Models read integers, not words. A tokenizer splits text into subword pieces and maps each to a fixed ID; an embedding matrix turns each ID into a long vector. Meaning lives in the geometry: related concepts sit near each other in that high-dimensional space.
-
Position and the residual stream. Plain attention has no notion of word order. Rotary position embeddings (RoPE) fix this by rotating the query and key vectors by a position-dependent angle, so the model tracks relative distance between tokens. The residual stream is the additive highway running through the network: each layer adds updates to a token's vector rather than overwriting it, which preserves early signal across a deep stack.
-
Attention, the communicator. This is where tokens exchange information. Each token produces a Query ("what am I looking for"), a Key ("what I offer to tokens looking at me"), and a Value ("the information I pass along on a match"). Dot products of queries and keys score alignment, softmax turns those scores into weights that sum to one, and the layer takes a weighted average of the value vectors.
The letters stay abstract until you watch them resolve a pronoun. Take "The server crashed because it ran out of memory." When the model processes "it", that token's query is effectively asking "I am a pronoun, who is my referent?" The key for "server" advertises "singular noun, subject of this sentence," and the two align far better than "it" aligns with "crashed" or "because." So the weighted average pulls mostly from the value of "server", and the vector for "it" leaves the layer carrying server-ness. No rule for pronoun resolution exists anywhere in the network; the behavior emerges because attention learned that this query-key geometry reduces prediction error. Every impressive long-range behavior (a variable used 3,000 tokens after its definition, a callback to the start of a conversation) is this same mechanism at longer distance.
-
The feed-forward network and MoE, the memory bank. While attention moves information between tokens, the feed-forward network (FFN) processes each token independently: expand the vector, apply a non-linearity, compress it back. This is where most stored factual knowledge lives. Attention is the conversation between tokens; the FFN is the library each of them steps out to consult. The parameter budget backs this up: in a standard transformer layer the FFN's two matrices hold roughly twice the parameters of the attention block (expand to 4x width and back costs about 8d² weights against attention's 4d²), so most of what you download when you download a model is the library. This is measurement, not metaphor. Geva et al. (2021, 'Transformer Feed-Forward Layers Are Key-Value Memories') showed that feed-forward layers behave as key-value memories: each key matches textual patterns from training and each value pushes probability toward the tokens that followed those patterns, with lower layers holding shallow patterns and upper layers semantic ones. Their opening line is the same parameter arithmetic: feed-forward layers hold two-thirds of a transformer's parameters. That arithmetic is also why MoE targets the FFN specifically: it is the biggest block, and the one whose lookups are per-token and therefore routable. Many frontier models extend it with Mixture of Experts: many parallel FFNs per layer, with a router sending each token to only a few (say 2 of 8), so total parameter count grows without the per-token compute growing with it.
-
Next-token prediction. At the final layer the last vector is multiplied by an unembedding matrix to produce a logit for every possible token; softmax turns those into a probability distribution. Temperature scales the logits first: low (0.2) sharpens toward deterministic output, high (1.4) widens variety, and top-k and top-p prune the candidate pool. Those three knobs define most of a model's apparent personality. A token is sampled, appended to the prompt, the KV cache stores the past keys and values so nothing is recomputed, and the loop repeats.
What interviewers probe next
- "Where do facts actually live?" Mostly in the FFN/MoE weights. Attention routes information; it does not store world knowledge.
- "Why bother with MoE?" It decouples parameter count from per-token compute: route to 2 of 8 experts and you get a huge model's capacity at a fraction of the inference cost.
- "What does the KV cache cost?" Memory grows linearly with sequence length times layers times heads, which is why long context is expensive and why batch size and context length compete for the same GPU memory.
- "Why RoPE over learned positional embeddings?" Relative positioning and cleaner extrapolation to sequences longer than those seen in training.
What sounds right and what the internals say:
| What people reach for | Why it fails | What to say instead |
|---|---|---|
| "The facts live in attention" | Attention computes weighted averages between tokens; it moves information, it does not hold it | "Attention routes, the FFN stores" |
| "MoE is a smaller model" | Total parameter count stays the same or grows; only the experts touched per token shrink | "Same capacity, fewer parameters active per token" |
| "It just predicts the next token" | True, and it names no mechanism, so it proves nothing | Trace the five stages, then say where the knowledge sits |
| "Temperature changes the model" | It rescales logits before sampling; the weights never move | "It reshapes sampling from a fixed distribution" |
Common mistakes
- Stopping at "it predicts the next token." True, but it describes no mechanism, so the interviewer concludes you read a thread, not a model card.
- Locating factual knowledge in attention. Attention is routing; the FFN is the memory.
- Reciting the softmax and dot-product formulas with no intuition for what Q, K and V are doing.
- Believing temperature, top-k and top-p change the model. They only reshape sampling from a fixed distribution; the weights never move.
Key takeaways
- Attention routes, the FFN stores: that one sentence proves you read the internals.
- MoE keeps the same per-token compute while growing total capacity by activating only a few experts per token.
- Sampling knobs (temperature, top-k, top-p) reshape a fixed distribution; they never touch the weights.
- Say "attention routes, the FFN stores" early, and cite the two-thirds parameter split if pushed; that pair is the difference between having read a model card and having read a headline.