Attention Is All You Need

In 2017 the best translation systems were recurrent networks. They read a sentence the way you read a news ticker: one word, then the next, carrying a running summary forward. That design had two costs. Training could not be spread across the words of a sentence, because step nine needs step eight. And a word at the start of a long sentence had to survive every step in between before it could shape a word near the end.

Eight researchers at Google removed the recurrence. Their model, the Transformer, lets every word look directly at every other word through attention, a learned and weighted lookup, and it does this for all the words at the same moment. Stack six of those layers, add a signal for word order, and that is most of the model.

It won on quality and on cost together, which almost never happens. The big model set a new English-to-German record of 28.4 BLEU, more than 2 points above every earlier system including ensembles. The small base model also beat every earlier system, and by the paper's own Table 2 it used about a twenty-third of the training compute of the previous best.

Nobody reading it in 2017 knew the same blocks would become GPT, BERT and the models behind today's chat assistants. That is why it sits near the top of this list: almost every paper after it assumes you know it.

Three questions you will get on a deployment come straight out of this paper.

"Why did doubling our prompt slow everything down so much?" The paper's Table 1 puts self-attention at O(n² · d) per layer for n tokens. Going from 8,000 to 32,000 tokens multiplies the attention score work by 16, while everything else in each layer grows by 4. Providers mostly bill per token, so customers feel this first as latency and context limits, not on the invoice.

"Why is the first word slow and the rest steady?" The paper trains on whole sentences in parallel, but its decoder writes one token at a time, each depending on the last. Reading the prompt is one parallel pass; writing the answer is a loop. Every latency conversation splits along that line.

"Isn't attention supposed to be the efficient option?" It was, for the inputs the authors had. Section 4 argues self-attention is cheaper than recurrence when the sequence is shorter than the model's vector width, which held for 2017 sentences against widths of 512 or 1,024. A 100,000-token context breaks that assumption by two orders of magnitude, and much of the serving work in this list exists to pay down the difference.