ppo
FDE interview questions tagged ppo, across every topic.
4 questions · 0 unlocked for you
Concepts behind "ppo"
The curriculum that explains the ideas these questions test.
Core
RLHF (Alignment)RLHF aligns a model to human preferences in three stages: supervised fine-tuning on demonstrations, training a reward model from human comparisons of outputs, then optimizing the policy with RL against that reward while a KL penalty anchors it to the base model. It shapes behavior and tone rather than facts, and FDE loops probe it because reward hacking and the KL anchor are where deployments actually go wrong.🧠 Foundations of LLMs & GenAISign in
Core
Policy Optimization: PPO and GRPOPPO and GRPO are the reinforcement-learning optimizers that turn a reward signal into model weight updates during RLHF and reasoning training. PPO uses a clipped objective and a KL anchor to keep updates stable and close to the reference model; GRPO drops the learned value network and instead normalizes rewards across a group of sampled answers. FDE loops probe these because the KL anchor, reward hacking, and GRPO's cost savings are where reasoning pipelines actually succeed or break.🧠 Foundations of LLMs & GenAISign in