Direct Preference Optimization (DPO)

DPO aligns a model directly from preference pairs (chosen vs rejected) without training a separate reward model or running an RL loop. It derives a closed-form solution to the same KL-constrained objective RLHF optimizes, turning alignment into a simple classification-style loss on the log-ratio between your policy and a frozen reference. FDE loops probe it because it is the practical default for preference tuning, and the trade-offs against PPO-based RLHF are where the judgment lives.