Speculative Decoding

Decoding is slow because each token needs a full forward pass through a huge memory-bound model. Speculative decoding has a small fast draft model propose several tokens at once, then the large target model verifies them all in a single forward pass and keeps the longest correct prefix. A careful accept rule makes the output provably identical to sampling from the target model, so you get lower latency for free, not an approximation.