Direct Preference Optimization: Your Language Model is Secretly a Reward Model

After supervised fine-tuning, the usual way to make a model behave the way people prefer has been RLHF, as in InstructGPT. Collect pairs of responses where a person said which was better. Train a reward model to predict those preferences. Then use reinforcement learning, usually PPO, to make the language model produce high-reward responses while a KL penalty keeps it close to the original model. It works, but it means training several models, sampling from the language model inside the training loop, and tuning a finicky optimizer.

DPO, from Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn at Stanford, shows the reward model and the RL loop are unnecessary. The objective RLHF optimizes has a known optimal solution: the best policy for a reward r is the reference policy reweighted by exp(r/β). Rearranging that equation expresses the reward in terms of the policy. Substituting it into the standard preference model (Bradley-Terry), an intractable normalization term cancels, and what remains is a loss you can compute directly from the language model's probabilities: raise the log-probability of the preferred response and lower the rejected one, both measured relative to a frozen reference model, with a logistic loss.

So DPO is a classification loss on preference pairs. No reward model to train, no sampling during training, and one main hyperparameter, β, which sets how far the model may move from the reference. The paper's subtitle sums it up: the language model is secretly its own reward model.

In experiments with models up to 6B parameters, DPO traced a better reward-versus-KL frontier than PPO on a controlled sentiment task, even PPO with access to the true reward. It matched or beat PPO on Reddit TL;DR summarization, generalized better to news articles, and was the only computationally efficient method that improved on the preferred responses in Anthropic's HH dialogue data.

DPO and its variants are now the standard way to do preference tuning on open models, including in fully open recipes like Tülu 3. If a customer wants a model to prefer one kind of answer over another, this is usually what you will run.

Preference data is easier to collect than demonstrations. People who cannot write the ideal answer can usually say which of two answers is better. Support leads, analysts and reviewers can label pairs quickly, and existing ratings or edits can sometimes be turned into pairs.

It fits in a normal fine-tuning pipeline. DPO is a loss function on pairs, so it runs with the same tools as supervised fine-tuning, including LoRA and QLoRA. That makes it practical on one or a few GPUs.

It still needs a good starting point. DPO assumes a reference model close to the data: the pairs should come from something like the model being tuned. When no such model exists, the paper first fine-tunes on the preferred responses. In practice: supervised fine-tuning first, then DPO.

β is the main dial. Lower β lets the model move further from the reference to match preferences; higher β keeps it close. Watch for over-optimization: preference tuning can make answers longer or more confident without making them better.

Evaluate with something other than the training preferences. The paper used GPT-4 as a judge and checked it against people; even then the prompt changed results. Use a held-out set and some human review.