Three stages, one KL leash. The reward-model-plus-PPO loop, the beta*KL term that keeps the policy from drifting off the reference, and how DPO collapses it into a single classification loss by treating the reward as beta*log(pi/pi_ref). The objectives, written correctly.
Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Three stages, one KL leash. The reward-model-plus-PPO loop, the beta*KL term that keeps the policy from drifting off the reference, and how DPO collapses it into a single classification loss by treating the reward as beta*log(pi/pi_ref). The objectives, written correctly.
The screen is whether the candidate can state the KL-penalized RLHF objective correctly (reward minus beta times KL to the reference policy) and explain DPO's key trick: the optimal RLHF policy has a closed form, so the reward becomes implicit (beta*log pi/pi_ref) and you optimize a pairwise logistic loss directly on preference pairs, dropping the separate reward model and the RL loop. The reserved follow-up: 'what is beta actually controlling' and 'when would you still prefer PPO over DPO.' Getting the loss form or the KL term wrong fails this.
No comments yet — be the first to share your approach.
