GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
← LLM & GenAI Fundamentals / 45
hardAnthropicOpenAIScale

Instruction tuning: win-rate keeps rising but factual QA accuracy drops. What's happening and the immediate fix?

Win-rate up, facts down is the signature of reward hacking: the policy is over-optimizing the preference proxy and drifting off the reference. The diagnosis (Goodhart on the reward), the immediate lever (tighten the KL/beta anchor), and the held-out factual gate that stops shipping the regression.

Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

Win-rate up, facts down is the signature of reward hacking: the policy is over-optimizing the preference proxy and drifting off the reference. The diagnosis (Goodhart on the reward), the immediate lever (tighten the KL/beta anchor), and the held-out factual gate that stops shipping the regression.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 548 remaining answers · ₹2,000 / $25

Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

Keep practising

Next in this trackDecoder-only, encoder-only, encoder-decoder: what is each good at, and why does generation use decoder-only?PremiumNext in this trackWhat is RoPE, why does relative position help, and why does it extend to long context better than learned absolute embeddings?PremiumNext in this trackPre-norm vs post-norm transformers: why do large models use pre-norm, and how does it affect training stability?Premium
FEDITOR'S NOTE

The screen is whether the candidate recognizes the diverging-metrics pattern as reward hacking / over-optimization (Goodhart's law on the preference proxy) and reaches for the KL-anchoring lever (beta) plus an early-stopping factual eval gate, rather than just 'collect more data.' The reserved follow-up: 'why does the preference model reward verbose, confident, sycophantic answers that are wrong' and 'how do you set the stopping point.' A candidate who only says 'add more preference data' misses the over-optimization mechanism.

DISCUSSION · 0

No comments yet — be the first to share your approach.