The Transformer, Intuitively

TL;DR: A transformer turns tokens into vectors, then uses self-attention so every token can pull in information from every other token, stacking that operation dozens of times to build meaning. It beat RNNs because attention processes a whole sequence in parallel instead of one step at a time, and the same all-pairs attention is why cost grows with the square of the sequence length.

The core move: attention

Before transformers, sequence models like RNNs read text one token at a time, carrying a running summary forward. That is inherently sequential, so it is slow to train and tends to forget what it saw long ago.

The transformer's idea is to drop the step-by-step recurrence and let every token look directly at every other token at once. For each token, attention asks "which other tokens should I pay attention to, and how much?" and blends their information accordingly. When the model processes "it" in "the trophy did not fit in the suitcase because it was too big," attention is what lets "it" pull meaning from "trophy" rather than "suitcase." Every token does this for every other token, which is where the all-pairs cost comes from.

How does a token decide whom to attend to? Each token produces three small vectors: a query, roughly "what am I looking for", a key, roughly "what do I contain", and a value, the information it will hand over if chosen. The score between one token's query and every other token's key becomes a weight, the weights are scaled to sum to 1, and the token pulls in the weighted sum of the others' values. "It" ends up with a high weight on "trophy" because the query it produced matches the key that "trophy" produced, and that is the entire mechanism behind the sentence above.

How a block is built

A transformer is a stack of identical blocks. Before the first block, once, each token becomes a vector plus a signal encoding its position: the token and position embeddings (attention itself has no inherent sense of order, so position must be added in). One block then does roughly this:

1. Self-attention. Each token gathers a weighted mix of information from the others.
2. Feed-forward network. A small per-token transformation that lets the model reshape what attention gathered.
3. Residual connections and layer normalization. The block adds its output back to its input (the residual) and normalizes, which keeps gradients stable so you can stack many layers without the signal degrading. The addition gives the training signal a direct path back through every block, so a deep stack still learns instead of fading toward the bottom.

Stack that block tens of times and you get a model that builds up meaning layer by layer, from surface patterns in the early layers to abstract relationships in the later ones.

Three shapes: decoder-only, encoder-only, encoder-decoder

The shapes differ in what each token is allowed to see:

• Encoder-only (the BERT family) lets every token see the entire sequence, past and future. That is great for understanding tasks like classification and producing embeddings, but it does not generate text left to right.
• Decoder-only (the GPT family, and most chat models) masks the future: each token can only attend to tokens before it. That constraint is exactly what makes the model good at generating the next token, one at a time. Most LLMs you interact with are decoder-only.
• Encoder-decoder (the original Transformer of Vaswani et al., 2017, 'Attention Is All You Need', and T5) runs an encoder over the input and a decoder that attends to both its own output and the encoder's, the natural fit for sequence-to-sequence tasks like translation. Worth naming so the taxonomy is not mistaken for a binary.

The quadratic cost, in words

With dense full attention, every token scores every position, so doubling the sequence length roughly quadruples the attention work. A 2,000-token prompt is not twice the cost of a 1,000-token one; it is closer to four times for the attention step. The count is simple: 1,000 tokens make a dense score matrix of 1,000,000 query-key entries per attention head, 2,000 make 4,000,000, and every layer and every attention head does it again. A decoder's causal mask hides future positions, so it uses about half of those entries, but the cost still grows with the square. This single fact explains why long context is expensive, why people obsess over the KV cache to avoid recomputing attention during generation, and why "just use a bigger context window" is never free.

Why interviewers probe this

They are checking whether you can explain a transformer without a whiteboard full of matrices, because as an FDE you will explain it to a non-technical buyer. The good answer is "tokens become vectors, attention lets each one look at all the others in parallel, you stack that many times, and the all-pairs attention is why long inputs cost more." The follow-up they hold back is "why is this faster to train than an RNN?" Answer: parallelism. The whole sequence goes through at once instead of one step at a time.

Common misconceptions

• "Transformers process text sequentially like RNNs." They process the whole sequence in parallel; generation is sequential, but the architecture is not.
• "Attention knows word order on its own." It does not. Position information must be injected through positional embeddings.
• "More layers always mean more understanding." Depth helps up to a point, but capability comes from the combination of scale, data, and training, not layer count alone.
• "Encoder and decoder are interchangeable." They differ in what each token can see; decoder-only masks the future, which is what enables generation.

Key takeaways

• A transformer embeds tokens, then uses self-attention so every token draws on every other, stacked across many blocks with residuals and normalization.
• It beat RNNs mainly through parallelism: the whole sequence is processed at once, making training far more efficient.
• Most chat LLMs are decoder-only, meaning each token attends only to earlier tokens, which is what enables left-to-right generation.
• All-pairs attention makes cost grow with the square of sequence length, the root reason long context is expensive.