The Llama 3 Herd of Models

The Llama 3 report covers a family of dense Transformers at 8B, 70B and 405B parameters (the results are for the Llama 3.1 release). Meta names three levers: data, scale and managing complexity. The architecture barely changes from Llama 2. The data and the scale change a great deal.

On data, Meta pretrained on about 15 trillion multilingual tokens after heavier deduplication, heuristic and model-based quality filtering, and a data mix chosen with small-scale scaling experiments: roughly half general knowledge, a quarter math and reasoning, 17% code and 8% multilingual text. A new 128K-token vocabulary packs 3.94 characters per token of English instead of 3.17. On scale, the 405B model used 3.8 × 10^25 FLOPs, almost 50 times Llama 2's largest model, and its size came from a scaling law built to predict benchmark accuracy rather than only loss. On complexity, Meta deliberately chose a dense model over mixture-of-experts, and DPO over PPO, to keep training stable.

Context grows to 128K tokens in six stages near the end of pretraining. Post-training repeats a round of reward modeling, rejection sampling, supervised fine-tuning and DPO several times. The 405B model lands close to GPT-4-class closed models on knowledge, code and math benchmarks, and Meta released its weights along with Llama Guard 3, a safety classifier. The infrastructure section is the most candid public account of what training at this scale involves: 466 job interruptions in a 54-day window, most of them hardware.

Llama 3 405B was the first open-weight model to sit near GPT-4 on most standard benchmarks, which changed what "keep it on our own infrastructure" could mean for a customer.

Serving 405B is a real commitment. At 16-bit precision the weights alone are about 754 GiB, more than an 8 × 80 GB node holds. Meta's answer is FP8 inference, which halves that. Long context adds more: a single 128K-token conversation needs about 63 GiB of KV cache even with grouped-query attention. The 70B or 8B model is usually the practical choice; the 405B is often best used to generate training data or to judge outputs.

The scaling-law method is reusable. Meta forecast benchmark accuracy before training by chaining two fits: compute to task loss, then task loss to accuracy. The same two-step idea works for deciding whether a bigger fine-tuning run will clear a customer's accuracy bar.

Reliability numbers set expectations. One unexpected interruption every 3.1 hours on 16,384 GPUs, more than half of them GPU faults. If a customer plans its own large training run, checkpointing, automated restart and failure diagnosis are not optional, and this paper gives the numbers to plan around.