Core
Policy Optimization: PPO and GRPO
PPO and GRPO are the reinforcement-learning optimizers that turn a reward signal into model weight updates during RLHF and reasoning training. PPO uses a clipped objective and a KL anchor to keep updates stable and close to the reference model; GRPO drops the learned value network and instead normalizes rewards across a group of sampled answers. FDE loops probe these because the KL anchor, reward hacking, and GRPO's cost savings are where reasoning pipelines actually succeed or break.
a free account unlocks the core curriculum tier · no card
Related concepts
Practise this in questions
LLM & GenAI FundamentalsGRPO vs PPO-based RLHF: how does group-relative reward let you drop the value model? →hard · PremiumLLM & GenAI FundamentalsWalk through the RLHF pipeline (SFT, reward model, PPO) and explain how DPO simplifies it. →hard · PremiumAI Security, Privacy & GovernanceTrain a red-team LLM with RL to find jailbreaks in your safety-tuned model, then feed the discovered attacks back into safety training. →expert · PremiumLLM & GenAI FundamentalsTrain a reward model for a coding agent from 100K noisy human scores biased toward short solutions. →expert · PremiumLLM & GenAI FundamentalsRL-train a 7B model to use SQL and a calculator with no human feedback on intermediate steps. →expert · PremiumLLM & GenAI FundamentalsWhat's the difference between a base model and an instruct model, and why would a customer care? →easy · Free