Tülu 3: Pushing Frontiers in Open Language Model Post-Training

A pretrained model predicts text; post-training turns it into an assistant. Labs like OpenAI, Anthropic and Meta publish results but rarely the data and recipes that produce them. Tülu 3, from the Allen Institute for AI and the University of Washington (Nathan Lambert, Hannaneh Hajishirzi and many others), is a complete open post-training pipeline: data, code, evaluation suite, intermediate checkpoints and a long report of what worked and what did not.

The recipe has four stages on top of Llama 3.1 base models at 8B, 70B and 405B. First, curate prompts for the target skills (knowledge, reasoning, math, coding, precise instruction following, chat and safety), from public datasets and new synthetic data, and remove anything that overlaps with the evaluations. Second, supervised fine-tuning on a mix of about 939,000 examples tuned skill by skill. Third, preference tuning with a length-normalized variant of DPO, on pairs built by sampling four responses per prompt from a pool of models, including the Tülu SFT model itself, and having GPT-4o rate them. Fourth, a new stage the authors call Reinforcement Learning with Verifiable Rewards (RLVR): PPO where the reward is 10 if a checker confirms the answer is correct (a math answer matches, an instruction's constraints are met) and 0 otherwise, with no reward model.

The results. At 70B, Tülu 3 averaged 76.2 on the development suite, above Llama 3.1 70B Instruct (74.1), Qwen 2.5 72B Instruct (72.8), Claude 3.5 Haiku (75.3) and GPT-4o mini (69.6). At 405B it was competitive with DeepSeek V3 and GPT-4o. At 8B the picture is mixed: it beat Llama 3.1 8B Instruct but averaged slightly below Qwen 2.5 7B Instruct.

The report's lasting value is its method: separate development and unseen evaluations, aggressive decontamination, careful data ablations, and an honest list of approaches that did not pay off.

Most customer model customization is a small version of what Tülu 3 does at scale: supervised fine-tuning on examples, maybe preference tuning, evaluated on what the customer cares about. This report is the best public playbook for doing that well.

Start from skills and evaluations, not data. The team first listed the skills they wanted and the benchmarks that measure them, then built data to move those numbers. For a customer: define the tasks and metrics, then collect data.

Check your data against your tests. Their decontamination found that some widely used open datasets contain large parts of standard benchmarks; one code dataset overlapped 70.7% with HumanEval. Training on that inflates scores without improving the model. Run the same check on customer data against the customer's evaluation set.

Keep an evaluation set you never tune against. The team held out an unseen suite and found they had overfit to some development benchmarks, notably MATH and precise instruction following. Hold out real cases nobody looks at during development.

Verifiable rewards suit enterprise tasks. If an output can be checked by code, such as a schema, a total, a format rule or a passing test, that check can serve as the reward for reinforcement learning, with no reward model. The report also shows the trap: when the only check is format, models learn to produce correctly formatted nothing.

Preference tuning gave the biggest single jump. At 70B, DPO raised the average from 72.6 to 76.2 and chat win rate from 26.3% to 49.6%. If a supervised fine-tune feels flat, preference pairs may be the missing step.