EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

Speculative decoding speeds up generation without changing outputs: a cheap draft guesses several tokens, the target model checks them all in one pass, and a probabilistic acceptance rule keeps the result exactly distributed as the target's. The speedup depends on how many guesses survive.

The EAGLE family drafts with a tiny head attached to the target rather than a separate small model. EAGLE fed the head the target's top-layer hidden states (the features just before the output layer) and trained it to predict the next feature, then used the target's output head to get a token. EAGLE-2 added a dynamic tree of guesses shaped by the draft's confidence.

The authors tried the obvious next step, more training data, and found EAGLE barely improved. Their diagnosis: requiring the draft to predict the target's next feature is an extra constraint on a model whose real job is predicting tokens. Removing it raised acceptance of the first guess but broke the second: from step 2 on, the draft has no target features for its own guesses and must feed back its own outputs, which now look nothing like the features it was trained on.

EAGLE-3 makes two changes:

• Training-time test. During training, the draft runs its own multi-step drafting and is trained on those self-generated inputs too, using adjusted attention masks so the extra steps are cheap. Acceptance stays flat across draft depth.
• Multi-layer feature fusion. With no need to reproduce the top layer, the draft can read a concatenation of low, middle and high-layer features from the target, compressed by a linear layer, plus the embedding of the latest sampled token, through a single decoder layer.

With these, more data keeps helping (a scaling trend EAGLE never showed). Trained on ShareGPT plus UltraChat (about 8x EAGLE's data) with target-generated responses, EAGLE-3 reached 3.0-6.5x speedups over standard decoding on Vicuna 13B, LLaMA 3.1 8B, LLaMA 3.3 70B and DeepSeek-R1-Distill-LLaMA 8B, and in SGLang it still added 38% throughput at batch size 64.

Speculative decoding is now a standard switch in SGLang, vLLM and TensorRT-LLM, and EAGLE-style draft heads are a common choice for open models. Knowing what drives their gains helps set expectations and decide when to train a custom head.

A draft head is model-specific. It reads the target's internal features, so each target model (and each fine-tune that shifts its features) needs its own trained head. Check one exists for the customer's model, or budget to train one.

Training data should look like production traffic. The paper trains on the target's own responses to ShareGPT and UltraChat prompts, plus math reasoning data for the reasoning model. Acceptance rates, and so speedups, depend on how close drafting data is to real use; code-heavy and template-heavy workloads accept the most.

It can help at moderate batch sizes. Speculation is usually assumed to hurt throughput under load. EAGLE-3 still gave 1.38x at batch 64 in SGLang because more guesses survive per verification; check where the crossover sits on the customer's hardware.