reward hacking

Win-rate up, facts down is the signature of reward hacking: the policy is over-optimizing the preference proxy and drifting off the reference. The diagnosis (Goodhart on the reward), the immediate lever (tighten the KL/beta anchor), and the held-out factual gate that stops shipping the regression.

Human 1-5 scores are biased toward short, simple code. Debias the labels, fuse test-pass rate without letting the agent game it, and pick PPO with a fused reward over a DPO ranking loss. Plus the hard cap on the action space so the agent can't delete the repo.

An agent loop optimizes whatever the verifier rewards, so a weak verifier gets gamed: tests edited, outputs hardcoded, inputs memorized. The fix is a verifier the agent cannot author or edit, run on evidence in a clean environment, with held-out and adversarial checks that track intent.