grpo

PPO needs a separate value network to estimate the baseline for its advantage. GRPO samples a group of answers per prompt and uses their mean reward as the baseline, so the advantage is just how a sample compares to its peers, no critic required.

Reward only on the final answer and learning crawls; shape intermediate tool calls and the model learns to spam SQL. Use outcome-grounded credit assignment, GRPO over a group of trajectories, and fine-tune a pre-aligned model so you're not teaching tool syntax from scratch.