Walk through the RLHF pipeline (SFT, reward model, PPO) and explain how DPO simplifies it.

The screen is whether the candidate can state the KL-penalized RLHF objective correctly (reward minus beta times KL to the reference policy) and explain DPO's key trick: the optimal RLHF policy has a closed form, so the reward becomes implicit (beta*log pi/pi_ref) and you optimize a pairwise logistic loss directly on preference pairs, dropping the separate reward model and the RL loop. The reserved follow-up: 'what is beta actually controlling' and 'when would you still prefer PPO over DPO.' Getting the loss form or the KL term wrong fails this.