Fast Inference from Transformers via Speculative Decoding

A large language model generates text one token at a time, and each step is a full forward pass. For big models at small batch sizes, that pass spends most of its time reading weights and the KV cache from memory; the arithmetic units sit mostly idle. Running the model on five positions at once costs about the same wall time as running it on one.

Speculative decoding uses that slack. A small, fast draft model (the paper calls it the approximation model) generates γ guesses autoregressively. The large target model then scores the prefix plus all γ guesses in a single parallel pass, getting its own probability for each position. Guesses are checked left to right with speculative sampling: keep a guessed token x with probability min(1, p(x)/q(x)). At the first rejection, discard the remaining guesses and sample a replacement from the normalized positive part of p − q. If every guess survives, the target pass also yields one more token for free.

The rule guarantees the output has exactly the target model's distribution, with greedy decoding, temperature, top-k or nucleus sampling, because all of them can be expressed as sampling from an adjusted distribution. Each target pass produces between 1 and γ + 1 tokens, so it is never slower in target passes than standard decoding. With a per-token acceptance rate α (the expected overlap Σ min(p, q)), the expected tokens per pass is (1 − α^(γ+1)) / (1 − α).

On T5-XXL with T5-small as the draft, the authors measured 2.3-3.4x faster decoding on one TPU-v4, with identical outputs and no retraining. Drafts about 100 times smaller than the target worked best; even a bigram model reached α = 0.2.

Latency at small batch sizes is where interactive products hurt: chat, coding assistants, agents that wait on each response. Speculative decoding is one of the few optimizations that cuts it without changing the model's outputs, which makes it easy to approve: quality can't regress.

It needs spare compute. The gain comes from idle arithmetic during memory-bound decoding. At high batch sizes, where the GPU is already compute-bound, verifying rejected guesses costs throughput. It's a latency tool first.

The draft decides the gain. Acceptance rate α and the draft's relative cost c set the speedup. A draft from the same model family, roughly 100 times smaller, is the paper's sweet spot; drafts that are too large accept more but cost too much (T5-large gave only 1.4-2.2x).

Free drafts exist. n-gram models, or copying text that already appears in the context, cost almost nothing to run. The paper proposes the copying idea for edits and rewrites, where the output repeats the input.